Surveillance: Achieving Safety, Security and Privacy using Machine Learning

by David Gamba

Privacy and bias concerns about surveillance devices are nothing new and these concerns are increasing every day as more and more devices are put into use. This topic has been around for quite some time and has even been touched on before, even in the music world. In 1984, a musician named Rockwell (also known as Kennedy Gordy, son of Motown founder Berry Gordy) released a hit song named “Somebody’s Watching Me”. In this smash hit, Rockwell sings that “I always feel like somebody’s watching me and I have no privacy”2. He repeats this refrain multiple times with references to people watching him constantly, even when he is at home. Additionally, he cites his home TV and phone as potential devices that are snooping on him and also notes that “people say I’m crazy”2 about these concerns and the fact that he feels his safety is at risk.

Fast forward 38 years later and Rockwell’s concerns are no longer viewed as irrational phobias, as billions of people are watched or monitored daily by a cadre of cameras and other devices. These devices include surveillance cameras, cameras on cell phones and moving vehicles, laptop cameras during video calls, live streams and at other times (whether noticed or not), thermal cameras taking temperatures, cameras set up to watch toddlers in cribs or elderly in their rooms, and many other types of sensors that are capturing and recording our movements and actions every day. According to Comparitech3, there are 770 million closed-circuit television (CCTV) surveillance cameras in use worldwide, with just over half of those deployed in China. Simple math shows that this means there are over 350 million surveillance cameras in cities outside of China. In London, England (UK) alone, there are over 625,000 cameras, or one for just about every 15 city residents3. With the continued growth of deployments being accelerated by the advent of things like COVID-19 compliance checking for adherence to mask and face-covering regulations, the number of CCTV surveillance cameras is expected to exceed one billion by the end of 20214.

Privacy is becoming a scarce commodity as all these peering lenses are watching people or recording their temperature and vital signs. This is being exacerbated by the ever increasing usage of in-home device assistants like Amazon Echo, Google Home, and Facebook Portal. It’s reported that there are over 128 million deployments of these devices across the US5 and these devices could potentially be listening and recording much more than users would like6. Many of these devices are connected to the cloud with information being recorded and sold without people’s knowledge, leading to an even bigger concern about privacy. In fact, governmental regulations are being put into place, trying to catch up with the advancements in technology given all these concerns over privacy. Even when these regulations are put in place, citizens are becoming more skeptical about who is capturing all this data and how it is being used.

No longer do people think it is crazy that someone is watching them. Conversely, today people are more concerned than ever about their loss of privacy on multiple fronts due to advances in technology. While individuals can choose to avoid social media and take other precautions to minimize intrusions into their privacy, they can’t completely opt out of society and avoid all these cameras. This is exacerbated by the recent advancements of facial and voice recognition technology enabled by machine learning (ML) used in many of these devices. There are also security concerns given that there is always the risk that the captured data will be stolen and used in nefarious ways.

Even though there will likely be more than one billion CCTV surveillance cameras in use throughout the world by the end of 2021, the number of cameras deployed in subsequent years is expected to increase. Though some are becoming accustomed to this, many people and municipalities are taking steps to minimize the impact to privacy. Both San Francisco and Boston have passed ordinances banning the use of facial recognition technology by several city agencies including the police. Recently, the European Parliament passed a new resolution that is targeted at preventing the use of facial recognition by law enforcement in public places. In addition to regulatory actions, there needs to be technical ways to minimize the impact on privacy. Smart cameras need to be developed with advanced ML technology that are capable enough to run Convolutional Neural Networks (CNNs) to capture the minimum amount of data needed for their function. Once in place, these smart cameras can perform local processing to eliminate the collection and storage of data, thereby making these devices less intrusive. For example, these ML-enabled smart cameras can have the intelligence to detect, identify, and grant or deny people access to enter buildings, stadiums, theaters, or other areas as appropriate and then immediately dispose of any captured information after the decision is made. Additionally, smart cameras with ML can have the intelligence to understand that there are certain images it should not capture and then prevent it from happening. For example, if an external camera used by a business is also viewing a window in a house, the smart camera can use ML to recognize that it’s a house window and that it should not be capturing information from that view. The camera would then prohibit the recording of any images of the window. This can accommodate for camera positioning challenges or mistakes — or even defeat intentional activity to capture prohibited information.

Clearly, the current approach using all these surveillance cameras connected to the cloud with cloud-based analytics is no longer an acceptable surveillance solution due to privacy and bias issues. Instead, localized embedded intelligence and analytics are critically needed to enable safety to be provided while managing data privacy. To achieve this requires very high performance ML at very low power. Though there is currently no viable purpose-built platform to address these needs on the market today, the encouraging news is that is developing ML system-on-chip (SoC) devices that can bring this added intelligence to cameras. These devices will make the cameras and sensors more capable and able to address many of these privacy and bias concerns. For example,’s Machine Learning System-on-Chip (MLSoC) devices support up to 50 Tera Operations per second (50 TOPS) at 5W for the ML (10 TOPS/W) and are capable of connecting to multiple cameras to perform sensor fusion to support image pre-processing → CNN on ML → post-processing in a single integrated device. These capabilities make the MLSoC device a great fit and enable the smart cameras to do image pre-processing to transform the video image inputs into a form that can then be processed through the multiple CNN models running simultaneously in a single MLSoC device to detect, identify, and validate a person trying to enter a building (Figure 1). If desired, the device can also check a person’s vital signs7 to ensure that they aren’t running a fever or are potentially infected with COVID-19 (Figure 2), even if they are wearing a mask or face covering. Both the identity validation and vital sign information can be discarded after checking and clearing the person to enter the building. Alternatively, the identity validation information can be saved and the vital sign information immediately discarded to allow accurate tracking of all who enter the building while not violating the Health Insurance Portability and Accountability Act (HIPAA)8 in the United States, General Data Protection Regulations (GDPRs) in the European Union, or other potential local government regulations prohibiting this. The smart cameras need to quickly process this information in real time by running multiple models in parallel in order to avoid long queues of people entering a building due to a lengthy screening process.

Figure 1: Flow for Local Facial Identification and Validation

Figure 1 shows the various steps a smart camera with’s MLSoC device can use for the detection, identification, and validation of a person trying to enter a building. This also demonstrates where the CNN networks need to be run in the MLSoC device in order to detect the faces of individuals and extract the facial features for the identification and validation of an individual. In this figure, the advantages of using an MLSoC device are seen whereby different blocks such as computer vision processors (green boxes), application processors (yellow boxes) and machine learning processors (pink boxes) can execute the various functions of the face detector, landmark detector and face feature extractor needed to perform the various tasks for detection, identification and validation.

As this example shows, the highly capable MLSoC device can locally perform sensitive functions to help address key application constraints and regulations required for privacy considerations. This example is just one of a multitude of different applications where smart cameras using the MLSoC device can cost-effectively address privacy and security concerns using a single chip that doesn’t need to be connected to the cloud.

Performing all these functions locally in the smart camera eliminates the undue cost burdens of sending all the data back to an expensive third-party cloud for processing. In fact, connectivity outages or even standard latency issues in dense urban environments can render a smart camera network that relies on the cloud inoperable. Also, many areas do not have easy access to cloud-based data centers so deploying smart cameras with cloud connectivity is infeasible.

Though using smart cameras with MLSoC devices capable of running multiple CNN models to do all the processing locally will not eliminate all the privacy, bias and security concerns about cameras watching us as we go about our everyday lives, it can make us feel a bit more at ease with these increasing intrusions into our lives.

For more information on, please visit

(1) Wikipedia
(2) Somebody’s Watching Me (Song by Rockwell) lyrics
(3) Surveillance Camera Statistics: Which Cities Have the Most CCTV Cameras?, Comparitech
(4) The Most Surveilled Cities in the World, U.S. News & World Report
(6) Amazon’s Alexa Never Stops Listening to You. Should You Worry?
(8) HIPAA Act of 1996 |