Panda Security, the 2009 product line and Collective Intelligence

Panda Security, the 2009 product line and Collective Intelligence

Introduction

The main change in Panda Securitys 2009 product line with respect to previous versions is Collective Intelligence. Next, we will explain the reasons for us to embark on this technological evolution.

Current situation

Lets start with the facts. Antivirus programs are subject to the following criticism: 

They are reactive with regard to the appearance of new threats.

They consume too many resources.

Being reactive is a consequence of the main technology included in all antivirus products: signature-based detection. Basically, this technology consists of incorporating signatures that allow the product to recognize malicious code in files. Even though this is a very old technology, used in the very first antivirus programs, it has been extremely effective. It has however a serious disadvantage: you need to have previous knowledge of the malicious code to be able to add its signature, test it and make it available to users.

With the huge amount of current malware in circulation, there is an issue that all antivirus companies have, but few are willing to admit:  if we only protect against the malware that we know, there is a high possibility that our clients are infected by new malware still unknown to us. Add this to the fact that malware has proliferated exponentially over the last few years, and you will be left with an unpleasant scenario. What is worse, all this is mainly based on hypotheses even though you can be sure the situation is just as described, you have to prove it scientifically to say it is really a problem to tackle.

This has been a known issue for many years, and, in fact, most anti-malware companies have been developing new technologies to detect the new malicious code that appears without having to depend on signatures. We are talking about old technologies, like heuristic technologies, and more recent ones like behavior analysis.

This takes us to the second criticism above: high resource consumption. Threats evolve over time and antivirus companies must adapt to this. Such is life, however, this might have some disastrous collateral effects. Viruses appeared and so did antivirus programs. All types of threats emerged and antivirus vendors started integrating all kinds of new technologies into their products to combat them: firewall, anti-spam, anti-spyware, heuristics, behavior analysis, content filtering, etc. The appearance of millions of new malware strains has multiplied the size of the signature file that needs to be loaded into the PC memory. Recently, most companies have started to add some new features to products, like backup copying or tune-up options. Even though we do this with the best of intentions, trying to offer users the best protection, we are overloading computers with mammoth tasks.

The birth of Collective Intelligence

Antivirus companies laboratories, besides analyzing new files and creating signatures to detect and disinfect them, develop technologies to detect new malware specimens. At present, PandaLabs has a system that classifies most files we receive: in 2007 we managed to add signatures for 94.4% of all new malware with this system. The question seems obvious: Why not integrate this into the antivirus programs we sell? Wouldnt we protect users better? Yes, we would, but the process capacity required would be so huge that it would be impossible to integrate it into desktop PCs. Bear in mind that this technology should be added to everything that is already on the computer, saturating it.

With all this in mind we started to sketch out the Collective Intelligence concept. What if we free the local PC from the calculation and decision-making work and leave all that effort in the hands of our servers? The idea was really tempting, and it could greatly increase our detection capacity: currently, when you look at a file in execution you examine it on the PC where it is running. Thats the information the antivirus will use to return a verdict on the files nature. Sometimes there is not enough information and it is not possible to take a final decision on whether a file is good or bad. However, if you have a central repository where you can correlate all evidence about a single file, or similar files, you will have much more information available to return a verdict.

Putting ideas into practice

In theory, all this is fantastic, however, it is when you try to put all this into practice that problems arise: What infrastructure is necessary? What information can be obtained while respecting each countrys laws? However, these small problems wouldnt stop us from reaching our goal. The idea was simply too good to dismiss so quickly.

Regarding the infrastructure, well, we must admit it is really impressive. At present we have over 150 servers dedicated to this task.

Regarding the information to obtain Well, this turned out to be very simple. During our internal meetings, someone suggested we would need to upload all files to our servers to analyze them there. Nothing could be further from the truth. To be able to classify a file all you need is an identifier and some information about it. If you analyze a 2 kb file you will need a few bytes of information, and the same applies to 10, 20 or 100 MB files. However, you dont even need to have a part of the file, but a checksum (https://en.wikipedia.org/wiki/Checksum). This way, the information to be sent is reduced to the minimum and we make sure we dont collect any personal data.

First proofs of concept and empirical demonstration of the problem

Last year we released a proof of concept of this technology: a Web-based antivirus that didnt have a signature file and took up less than 400KB. It just performed on-demand scans of files running in memory, and even as such, it was greatly received. After some time, we decided to study the data obtained and check if the capacity of this new concept lived up to our expectations. The data obtained was very good but also very worrying. Let me explain this.

On every computer where a scan was performed, a query was made to the Windows Security Center to find out if there was an antivirus installed, if it was active or not, if it was updated, etc. When it came to analyzing the data, we separated the computers that didnt have an installed, activated, up-to-date antivirus. We also started to draw up data by antivirus vendor. What was the result? Clients of ALL antiviruses (Panda included) were infected.  We published a white paper (https://research.pandasecurity.com/blogs/images/wp_pb_malware_infections_in_protected_systems.pdf) with all the data, but here is a summary of the most interesting conclusions:

- Only 37.45% of all PCs had an active, up-to-date antivirus.

- Over 23% of scanned computers with an active, up-to-date antivirus had malware in memory. By malware we refer to all kinds of threats: viruses, Trojans, worms, spyware, etc.

- These are the infection percentages by antivirus company:

CA:  23,32%

McAfee: 24,18%

Panda: 15,54%

Symantec:  22,20%

Trend Micro: 17,08%

The companies appear in alphabetical order as this is not a ranking. The study contained a total of 30 companies, all of them with infection rates between 12-13% to 30% approximately. If you want to know my opinion about rankings you can read the article I posted a few weeks ago (https://pandalabs.pandasecurity.com/archive/Mine-is-bigger-than-yours_2100_.aspx).

Integration of Collective Intelligence in our products

After all this the path to follow was very clear, so we started to apply these technologies to the 2009 product line that we were developing. What is the difference between these products and the previous ones? The most revolutionary change is the integration of Collective Intelligence. What benefits does it offer?

- Optimized signature file (which is loaded into memory to detect known malware).

- Connection to Collective Intelligence when scanning, to maximize malware detection.

The figure below shows the systems basic functioning:

On top of all this, there is TruPrevent 2.0: the fusion of TruPrevent 1.0 with Collective Intelligence The first version of TruPrevent technologies was designed to complement antiviruses, helping them to stop unknown malware. To do this, these technologies analyzed the behavior of all the programs that bypassed the antivirus scan, blocking those that carried out harmful actions and turned out to be unknown viruses.

TruPrevent 1.0 scanned all potential threats using different techniques, carrying out complementary, in-depth analyses at different levels of the infrastructure. Technologically speaking, TruPrevent consists of two main components: behavior-based analysis and blocking.

Panda released this technology in 2004. Since then, four years have gone by of constant innovation and improvement to finally release this new version. This versions engines (heuristic, behavior scanner, etc.) have been adapted to the current malware by using new sensors, etc. Also, every time they detect something they launch a query to the cloud for information about the file. TruPrevent is therefore more effective when it comes to detecting malware, producing almost zero false positives.

To sum up, TruPrevent has been adapted to the current malware situation, characterized by an avalanche of new malicious code every day. This new technology detects and blocks a larger number of strains with less consumption of PC resources. This is due to the fact that most information is hosted on Pandas servers, not on the users PC.

Questions and answers

When new technologies are introduced on the market, there are always questions that need to be answered. Here is a summary of the most frequently asked questions:

Q: What is an optimized signature file?

A. PandaLabs uses a network of over 4,000,000 sensors to know which malware is currently in circulation and select the necessary signatures to protect PCs against genuinely active malware.

For example, a proof of concept of a malware sample created in 2000 which has never been detected as being active by our sensors will not be included in the signature file. However, a virus that appeared in 1995 but has been active will be included in it.

In any event, Collective Intelligence includes millions of signatures not only of malware, but also of millions of known good files, which provides us with the largest detection capacity.

But there is more: TruPrevent proactive technologies, even though they benefit from communicating with Collective Intelligence, dont need to be connected to the Internet to detect and stop unknown malware.

Conclusion: Triple protection Signatures + TruPrevent + Collective Intelligence, taking up less resources.

Q: Why is it necessary to keep using the signature file, if Collective Intelligence includes all of the signatures?. Couldnt you eliminate it?

A. Collective Intelligence requires an open Internet connection. Even though it is true that over 99% of todays infections come from the Internet, there is a small risk window when the Internet connection is not open but you introduce files into the system using a storage device, from a CD to a USB device. Thats why we keep using the signature file that contains all active malware, to keep you protected from those attacks.

Q: As my computer needs to keep in touch with Collective Intelligence, wont this affect my Internet connection due to the bandwidth required to perform queries?

A. No. As I explained before, only a few bytes are needed for each file. The answer to the query will also take a few bytes as well. Also, this is a smart system which creates a local cache to avoid querying about the same files over and over again.

Q: I understand that thanks to Collective Intelligence the product will use fewer resources. However, wont I notice a certain slowdown in my computer due to the communication between my system and Collective Intelligence every time a new file arrives?

A. No. The system has been designed to increase detection capacity and reduce resource consumption. The antivirus products that dont use this technology are more resource intensive.

Q: So then, am I completely protected now?

A. No. I dont want to be ambiguous about this. NO. There is no technology today that can give you a complete protection guarantee. Having said that, however, my personal experience tells me that this is the technology that offers the best protection on the market.

Near future

And now what? The truth is that the original idea we had is not totally reflected in the 2009 products, as we are very ambitious. I cannot give you all the details right now, but lets say our idea is something more like Nanoscan: a tiny, extremely light antivirus. We are convinced we are working in the right direction: Trend Micro has recently announced it is going to release products that will connect to the cloud, and I am sure other competitors will join us over time.

Finally, Collective Intelligence has another benefit that I havent mentioned yet but which many of you will have already guessed: if we create new malware classification and detection systems, we will be able to apply this technology to all our products transparently, as the new technologies will be applied to our Collective Intelligence systems instead of to the products installed on our clients PCs.