ROXANNE – Real time network, text, and speaker analytics for combating organized crime – is the most recent EU-funded collaborative project that the HENSOLDT Analytics Research Team successfully completed. After three years of intense work, the partner consortium celebrated the project’s conclusion in November with a final conference in Paris.
We asked the Research department colleagues about their experience of working on ROXANNE – and what made this particular project especially important for promoting the case of speech and language technology in collecting actionable intelligence.
ROXANNE was a truly collaborative project, also within HENSOLDT Analytics. We would like to acknowledge every member of the team, who contributed to making the project a success: Erinç Dikici, Dorothea Thomas-Aniola, Miroslav Janosik, Maximillian Schürz, Gerhard Backfried, Katja Prinz, Antonio Ramos, and Aneta Pawlik.
Can you briefly introduce the ROXANNE project?
ROXANNE is a project funded under the EU Horizon 2020 Work Programme for research and innovation. It began in 2019 and is now reaching its end. The final project conference took place on the 29th of November at the Capgemini’s Campus Cyber in Paris, which we of course attended to present the results of our work and to celebrate with the entire consortium and stakeholders!
The project itself focused on several of our areas of expertise – utilising speech and language technology to collect intelligence – and combining these with network analysis. ROXANNE’s primary objective was to create a mix of technologies to assist law enforcement and intelligence agencies across Europe to track and uncover networks of criminals. What was so special about ROXANNE is that speech and NLP technologies were complemented with link analysis, e.g. techniques to evaluate (hidden) relationships/connections between nodes. This resulted in a platform which enables LEA’s to efficiently investigate whole networks of criminals across a variety of modalities and channels that use voice communication.
This is quite a complex task. How did the consortium plan to achieve the goals of ROXANNE considering technological and organisational limitations, for example, how the research part ties into real-life investigations; the topics of data privacy and ethics, or the problem of not having access to real data from investigations?
It definitely was a challenge, but the composition of the consortium was one of the key ingredients to putting the project proposal into reality. We had a good mixture of research partners – our coordinator, IDIAP Research Institute, the L3S Research Centre of the University of Hannover, the University of Saarland, Brno University of Technology, Transcrime Joint Research Centre on Transnational Crime, Trilateral Research, Aegis IT Research, and the Center for Security Studies (KEMEA). We also had professional tech companies on board, such as ourselves, Capgemini, Airbus, ITML, Phonexia, and ADITESS. And what I think was incredibly important to the success of the project is, that we had several law enforcement and intelligence agencies in the consortium, who helped us validate whether the system we were working on was in fact meeting their expectations and usable. INTERPOL, the forensic institutes from the Netherlands (NFI) and Lithuania (LTEC), the Romanian Center for the Coordination of Scientific Research, and police departments from the Czech Republic, Croatia, Greece, Ireland, Israel and Northern Ireland. Furthermore, ZITIS – the German Central Office for Information Technology in the Security Sector also joined the consortium during the runtime of the project. Overall, this was a sizeable consortium of 25 partners that were very well matched for the requirements and scope of the project.
Getting our hands on data for training and evaluation was a very challenging task. Due to prior experiences, we were well prepared. We used data from public resources and created a sizeable (and realistic) corpus of simulated data. Based on input from police organisations, volunteers re-enacted actual cases, pretending to be part of criminal networks. All recordings took place under realistic conditions and using actual interception infrastructure, yielding a dataset similar to what can be obtained by wire-tapping of phones. In addition to this activity, consortium partners from law enforcement agencies shared their investigative experience with the technical partners to improve the ROXANNE platform and technologies.
Of course, we paid close attention to privacy and ethics issues and no real personal data was included. Throughout the project, it was of paramount importance to us to respect privacy, and ethical and legal aspects.
What was HENSOLDT Analytics’ role in the project?
HENSOLDT Analytics led the work package, which dealt with data management, and participated in several other work packages. On the one hand, this entailed the creation of an inventory of data resources, on the other hand, it comprised work on speech, text and video data analysis. Identifying, finding, and creating datasets which can be used for training and evaluation was a challenging piece of work. Since many technologies are based on ML (machine learning) methods, providing the right kind of data for training is essential. And so is the right kind of data for evaluation – which should reflect realistic conditions. If there is a mismatch between these conditions, the models would not stand the test of real-life cases. Furthermore, there was a lot of work involved in extending and adapting technologies to fit the requirements of the ROXANNE use cases (network analysis, natural language processing, speaker identification). For example, we worked on developing a model for updating vocabularies semi-automatically for automatic speech recognition (ASR).
Why was it important that HENSOLDT Analytics was part of ROXANNE?
At HENSOLDT, we already have the know-how and the speech technology for the purpose of investigations or situational awareness. However, it is always crucial to stay in touch with the latest developments in technologies and keep close contact to research institutions, end-users and stakeholders. The different areas of expertise and backgrounds of consortium members all add to the mix and help us extend the overall picture and come up with innovative approaches.
What unexpected challenges did you face throughout the project? Did you manage to overcome them?
The biggest challenge – as in many projects involving LEAs – was probably the fact that we could not obtain any real data from the end-users to adapt and evaluate our technologies as well as the ROXANNE solution, due to legal, ethical and privacy constraints. This led us to collect our own data, the ROXANNE Simulated Dataset (ROXSD), which is a multimodal, multilingual, and fully annotated dataset of communication in organized crime. It is the first of its kind and allows benchmarking of several audio-visual technologies. We hope to share ROXSD with the research community soon.
You and the team participated in ROXANNE field tests to collect the feedback from the end-users of the ROXANNE platform. What were your lessons from these events with the law enforcement officials?
The three field tests during the project were a great opportunity to understand the law enforcement agencies’ needs and operational capabilities. We observed once again how important speech and language technologies could be in their daily work, and how the combination of different technologies could improve their investigation performance. Furthermore, we learned about the possibilities and limitations of the use of open-source technologies for their work.
What surprised you the most in those 3 years you spent with ROXANNE?
We’ve taken part in many research projects over the years. However, ROXANNE was the first one where end-users joined the project during the runtime – in the case of ZITiS – or associated them with the project without becoming official members – in the case of a German Criminal Police organisation. Both cases demonstrated the relevance ROXANNE has for LEAs. The addition of a new partner as well as the association of another has led to many additional links into LEAs across Europe.
Having completed ROXANNE, what will happen with the outputs of the project?
During work on the project it became clear that no adequate dataset for training and evaluation for the combined set of tasks addressed by the ROXANNE project existed. This resulted in the creation of the ROXSD, in which the story is inspired by (anonymised) actual cases of one of the LEA partners. Recordings using real interception equipment were made with voice actors during several workshops. The resulting dataset was curated and prepared carefully and now presents a very unique dataset for demonstration and evaluation purposes. This dataset will be made available for research purposes and will be a valuable asset for future research.
Regarding the technologies, the consortium adopted an open-source approach leading to the creation of the Autocrime platform. Autocrime comprises a suite of technologies and workflows as well as a GUI for visualisation and interaction. It will also be made available to European LEAs and other stakeholders and allow them to demonstrate and extend their capabilities. During a recent workshop with numerous LEAs (not involved with ROXANNE) both, the dataset as well as the platform, created considerable interest.
What was the key takeaway for you as a team of researchers from this project?
The rapid changes in technology reflect themselves also in the methods used by criminals. Criminals are known to be among the early adopters of new technologies and even drive innovation. Investigators and investigation capabilities have to keep up with such changes (as do legal and procedural frameworks). This requires scalable multi-modal, multi-lingual processing capabilities as well as a multi-disciplinary approach. We believe that the ROXANNE project has taken an important step towards creating a modular and flexible platform with many technologies included, which the LEAs as well as all researchers in the field can benefit from. As HENSOLDT Analytics, we aim to continue participating in research partnerships like ROXANNE and to offer our state-of-the-art technologies for the building of a safer world.
The ROXANNE project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement № 833635.
