Couple of days ago when I was assisting a customer, I recommended that they follow the National Institute of Standards and Technology’s (a.k.a. NIST) Information Technology Laboratory’s Computer Security Division’s National Vulnerability Database’s (a.k.a. NVD a.k.a. that place that publishes the CVEs) data feeds. (Apologies for the bad intro)
So, these are RSS feeds that “contains the most recent CVE cyber vulnerabilities published within the NVD”
Unfortunately, I saw a notice at the top of the page, which got me really worried. It says
In September 2023, the NVD plans to retire all legacy data feeds while guiding any remaining data feed users to updated application-programming interfaces (APIs).
Usually, I’d panic and start ranting on my blog, but this is the NVD we’re talking about. They are a US government project that has been doing a lot of good and they are sponsored by the CISA, an agency that does many good things not just for US citizens, but citizens of our planet.
I started digging to understand what exactly is going to be retired and most importantly, why?
The NVD has made an amazing change timeline that has the following
The NVD plans to retire the RSS data feeds. The NVD plans to enable reCAPTCHA across all webpages and to retire webpages intended to support web scraping (e.g., Full Listings) before its APIs existed.
Okay, NOW I’m worried.
I’ll break this into two parts.
Why we need RSS feeds
You see, the internet relies on RSS, and I’m not just saying that because most of my audience uses RSS daily. The reason is much deeper than that.
As Dave Winer blogged a month ago
RSS is a thing like roadways and paths of rivers, they change very slowly. Think about qwerty keyboards. That’s what we’re talking about here. Agreements between products to interop. RSS is just like the gauge of rails, or always driving on one side of the street. A convention that makes progress possible. #
There are three products/protocols that I use daily, it’s Slack (for work), XMPP (for friends and family) and Telegram (for Armenian tech communities).
There are specific things that I should deliver for all these and that is messages, alerts, notices.
For my work, I should be able to get news if there’s a security issue on FreeBSD, because we use that. For friends and family I should deliver notices if there are any issues or upcoming maintenance to our servers. For my telegram communities I should update them if we’re having any new meetups, events, podcasts.
But, instead of writing a software that fetches, parses, analyzes and does something-something to these messages, I use RSS! FreeBSD has an RSS feed for Security Advisories. All I do in Slack is /feed subscribe https://www.freebsd.org/security/feed.xml and now, every time there an SA for FreeBSD, I get notified in Slack.
For friends and family? I have Huginn agent that parses RSS and send an XMPP message. For Armenian tech communities? I read a website’s RSS and a bot posts it in a group.
You get the idea.
RSS is all about “things working together”, there is no need to write a specific piece of for that specific thing.
And for years, I’ve relied on NVD’s RSS data feed to notify customers, tell them what to upgrade, if they need to upgrade and why to upgrade.
These RSS feeds are part of my professional life, a way for me, and people like me to know if we should be in panic mode or not.
So…
Okay, now what?
I believe in communication. I was very sure that my questions will be answered by the NVD, so I sent a message!
Greetings dear NVD team, NIST team and Computer Security Division,
While browsing your website, I have noticed the following change:
> In September 2023, the NVD plans to retire all legacy data feeds and the 1.0 APIs.
This became very disturbing, as many companies (including mine) rely on the data feed provided by NIST’s NVD.
I have two questions:
1) Is there *any* chance to keep the RSS feeds?
2) Is it okay if others (i.e. I) generate an RSS feed from your new API, if your final decision for q#1 is no?[ . . . ]
If I may, that being said, I’m sure there’s a good reason, so my other question is:
What are/were the technical issues with RSS? Could it be bypassed or hacked around?Thank you for all the work that you do, and thank in advance.
Kind regards,
I was right! They did answer all of my questions! I got a reply yesterday, here it is.
1) Is there *any* chance to keep the RSS feeds?
We have no plans to continue providing the RSS feeds located at
https://nvd.nist.gov/vuln/data-feeds#RSS
https://nvd.nist.gov/feeds/xml/cve/misc/nvd-rss.xml
https://nvd.nist.gov/feeds/xml/cve/misc/nvd-rss-analyzed.xmlAdditionally, it is important to point out that per our announcement at https://nvd.nist.gov/general/news/change-timeline, the RSS feeds will be retired in March, not September. If you were not aware of these announcements we highly advice joining the NVD Google Group to stay better informed (https://groups.google.com/a/list.nist.gov/g/nvd-news).
2) Is it okay if others (i.e. I) generate an RSS feed from your new API, if your final decision for q#1 is no?
All NIST publications are available in the public domain. Organizations seeking to automate the retrieval of NVD data should use the NVD’s Application Programing Interfaces (APIs).
Services which utilize or access the NVD are asked to display the following notice prominently within the application: “This product uses data from the NVD API but is not endorsed or certified by the NVD.” You may use the NVD name to identify the source of the data. You may not use the NVD name, to imply endorsement of any product, service, or entity, not-for-profit, commercial or otherwise. For information on how to the cite the NVD, including the database’s Digital Object Identifier (DOI), please consult NIST’s Public Data Repository.3) What are/were the technical issues with RSS? Could it be bypassed or hacked around?
The RSS feeds were considered to be overly simplistic and underutilized, they were determined in scope of retirement for these reasons as part of a larger effort to consolidate our output formats as we move towards the APIs. If you would like to submit a user story explaining the benefits and needs that the APIs currently do not meet we would pass that along to the development team for consideration in the future.
Okay. I agree! RSS is very simplistic, but that’s the point! it’s supposed to be simple. I mean, it’s simple enough that podcasts are RSS feeds.
And to be clear, I DID check the NVD’s new Vulnerabilities API, it’s awesome, it’s nice, it’s documented very well, kudos to the team, they did an amazing work, I’m sure it wasn’t easy. It has, for sure, more features than RSS could provide.
What to do about it?
I understand that the NVD is pushing the REST API, and I also understand why. But I really don’t want to write a “wrapper” for every service and technology that I use.
Here are my two questions.
- Will systems break because of this? Are you using these feeds? Do you rely on them for yourself or your organization?
- Will there be an interest by the InfoSec community to write a wrapper that generates a new RSS feed from NVDs new API?
Personally, if there’s an interest or not, I will be stopping everything I’m working on to create this NVD-to-RSS generator, as I very much rely on it. It will be open-source, obviously. What I should build is a drop-in replacement, where you change the feed URL, and everything works like before. (Well, I have to finish my other open-source commitments first, then I should work on this 😀 hopefully it wont take long.)
I would like to thank the NVD for keeping these feeds for all these years and congratulate them for their new APIs, I’m sure many good things will come out of these APIs.
And thank you for reading 🙂
That’s all folks…
