CyberHULL

Client employed us to reverse-engineer and re-create an existing back-end solution provided by external vendor with the goal to gradually replace it. The system had to handle incoming authentication, authorization, geolocation checks and other various user data and content management requests, as well as securely store incoming data. The goal was to improve reliability and performance with the priority on the former.

Applications hosted on on-premises infrastructure needed to be migrated to the cloud, necessitating a review of existing solutions, architectural updates, and the establishment of secure, scalable private communication between services across different AWS accounts. Furthermore, the project required the development of a reliable multi-region failover solution, ensuring equal traffic distribution and minimal downtime during failovers.

Absence of proper documentation necessitated reverse engineering of the existing solution. After initial analysis and in consultation with the client C#, .NET Core and Apache Cassandra were chosen as the primary technologies.

.NET Core allowed us to use technologies familiar to the client, while enabling dockerization and avoiding Windows. Specific third-party libraries lacking in .NET Core ecosystem were decompiled, reverse-engineered additionally optimize¬d and re-implemented on .NET Core.

Implementation of proper JWT-based authentication and authorization allowed to significantly cut down on cache and database usage, while increasing both speed and reliability solutions operation. Implementing popular HTTP hardening methods significantly increased system’s stability, including its ability to stay operational during outages of critical third-party services. Where appropriate, inter-service communication was conducted using message queues.

Data modeling results confirmed that using NoSQL database was more appropriate for the solution. Apache Cassandra was chosen based on client’s preferences. This was later migrated to AWS DynamoDB live with no downtime incurred as client decided to shift to managed solutions. When creating data models for both databases extra care was taken to adhere to all known best practices of corresponding solutions, such as hot partitions, tombstone management, GSI overload. Both databases provided out-of-the-box support of geographically distributed clusters synchronization. After the migration to DynamoDB, multithreading-related issues were observed when handling certain request patterns. After communicating it to the AWS development team they fixed in the follow up releases.

Additionally, client required implementation of a custom centralized configuration storage, which in turn necessitated enhancement and expansion of existing .NET Core libraries to enable real-time synchronization.

Application-level monitoring was introduced using technologies already employed by the client. The metrics were continually updated based on issues found and identified.

Implementation of backend automation integration test suite allowed to ensure both functional and performance consistency throughout the follow up development of the solution.

The resulting system’s GDPR adherence was ensured based on client’s request.

Mission-critical reliable and performant system serving the requests of entire client's user base. Zero downtime incurred throughout the years after system’s rolled into production. System can stay operational despite critical third-party system outages. System is easily scalable at the datacenter and geographical levels with little to no additional developer involvement.

This case study delves into the development of a robust pipeline designed to process LIDAR scans and panoramic photographs of city streets for seamless integration into navigation maps. The raw data, acquired from a single scanner car ride, poses challenges due to its massive size, noise, and inherent inaccuracies. The objective is to simplify the data and align it with known landmarks to achieve acceptable precision, considering the scale of multiple car rides per day. The pipeline, implemented in Java Enterprise and Hadoop, leverages highly parallel data processing to address these complexities.

Raw data from a single scanner car ride is voluminous, noisy, and inherently inaccurate. Achieving acceptable precision requires simplifying and aligning the data with known landmarks, especially considering the scale of multiple car rides conducted daily.

The pipeline addresses the challenges through a strategic implementation:

Java Enterprise and Hadoop: Leveraging Java Enterprise and Hadoop, the pipeline is designed for highly parallel data processing, enabling efficient handling of massive datasets.

Data Simplification and Alignment: Advanced algorithms are employed to simplify the raw data and align it with known landmarks, ensuring improved precision in the final output.

The implementation of the pipeline yields several positive outcomes:

Massive Data Processing: The pipeline efficiently processes massive and complex datasets derived from multiple scanner car rides, ensuring scalability and reliability.

Noise Reduction and Accuracy: Advanced algorithms applied to the raw data simplify and align it, significantly reducing noise and enhancing overall accuracy.

Highly Parallelized Operation: Leveraging Hadoop for parallel processing ensures the scalability and efficiency needed to handle the large-scale data generated by numerous car rides daily.

Seamless Integration with Navigation Maps: The processed data seamlessly integrates into navigation maps, contributing to a more accurate and reliable mapping experience.

In conclusion, the LIDAR and panoramic photograph processing pipeline represents a pivotal solution in achieving precision in urban mapping. The strategic use of Java Enterprise and Hadoop for highly parallel data processing, coupled with advanced algorithms for data simplification and alignment, ensures the seamless integration of accurate information into navigation maps despite the challenges posed by massive, noisy, and inaccurate raw data.

This case study explores a comprehensive project involving the migration of applications from on-premises to the cloud, AWS account separation, and the establishment of a reliable multi-region failover solution. The initiative required a meticulous review of existing solutions, architectural updates, and the implementation of Blue-Green/Canary deployment strategies, multi-region failover plans, and advanced monitoring solutions. Additionally, a solution for private communication between services across different AWS accounts was developed, focusing on scalability, reliability, and security. The study also covers the development and deployment of a robust multi-region failover solution, ensuring traffic distribution across regions, and thorough documentation and monitoring.

Applications hosted on on-premises infrastructure needed to be migrated to the cloud, necessitating a review of existing solutions, architectural updates, and the establishment of secure, scalable private communication between services across different AWS accounts. Furthermore, the project required the development of a reliable multi-region failover solution, ensuring equal traffic distribution and minimal downtime during failovers.

1. Migration and Environment Separation

Review and Architectural Update: In-depth review of existing services, interactions, and architectural updates following the latest security and reliability conventions, including Blue-Green/Canary deployment strategies and multi-region capabilities.

IaC Development: Creation of reusable and scalable Infrastructure as Code (IaC) to introduce new environment types or update specific properties without affecting other environments.

AWS Account Separation: Provisioning of environments across separate AWS accounts, rigorous testing, and close collaboration with service developers and organizational teams.

Deployment Strategies: Setup of build, deployment, Blue-Green/Canary strategies, and multi-region failover plans.

Switchover Process: Active participation in switching environments from development to production, ensuring minimal downtime for end clients.

2. Advanced Monitoring Solutions:

Monitoring Setup: Implementation of advanced monitoring solutions, including metrics, dashboards, and alerts, to track service health and availability.

Documentation: Detailed documentation of service configuration, endpoints, environment setup, service workflows, and dependencies, with architecture diagrams depicting the setup in various scenarios.

3. Private Communication Solution:

Scalable, Reliable, Secure: Development of a solution for private communication between services across different AWS accounts, emphasizing scalability, reliability, and security.

Traffic Distribution Control: Implementation of plans for controlling traffic distribution across regions, factored for various failover scenarios.

Thorough Documentation: Comprehensive documentation of the solution, including workflow descriptions and architecture diagrams for different infrastructure states.

Application Updates: Update of application documentation to reflect changes in setup.

4. Multi-Region Failover Implementation:

Development and Deployment: Creation, testing, and deployment of a reliable multi-region failover solution across all application stacks.

Traffic Distribution Control: Creation of separate plans for each application to control traffic distribution across regions during failover scenarios.

Documentation Update: Thorough documentation of the implemented solution, including workflow descriptions and architecture diagrams.

Collaborative Monitoring: Joint efforts with other teams to establish monitoring solutions for observing traffic distribution and application states during failovers.

Successful Migration: Successful migration of applications from on-premises to the cloud, with environments separated across multiple AWS accounts.

Seamless Switchover: Efficient switchover from development to production with minimal downtime for end clients.

Private Communication Solution: Implementation of a secure, scalable, and reliable solution for private communication between services.

Multi-Region Failover: Development and deployment of a robust multi-region failover solution, ensuring equal traffic distribution and thorough documentation of the infrastructure in various scenarios.

In conclusion, this case study highlights the success of a strategic initiative involving application migration, environment separation, private communication solutions, and the implementation of a reliable multi-region failover strategy. The meticulous planning, execution, and collaboration with various teams resulted in a seamless transition to cloud infrastructure and enhanced operational resilience.

This case study explores the development of an iPad application for flor.com, an innovative rug and carpet manufacturer. The e-commerce app, specializing in selling rug and carpet tiles, incorporates a unique in-app designer for creating custom carpet designs. The application addresses the challenge of selling carpets in 50x50 cm tiles by providing customers with an intuitive solution for experimenting, creating, previewing, and ordering personalized carpet designs. The in-app designer allows users to create carpets of any shape and size, using any combination of available tiles. The gallery showcases predesigned carpets, while the general shopping section facilitates conventional ordering of new or replacement tiles and supplies.

The challenge was to provide customers with a seamless and innovative way to shop for rug and carpet tiles, especially when sold in 50x50 cm tiles. Customers needed a solution to experiment with, create, preview, and order custom carpet designs directly from an iPad application.

The iPad application addressed this challenge through various features and functionalities:

In-App Designer: The application includes an intuitive in-app designer that enables customers to create custom carpets of any shape and size, utilizing any combination of available tiles.

Gallery of Predesigned Carpets: A curated gallery provides customers with a selection of predesigned carpets for inspiration and quick ordering.

General Shopping Section: In addition to the in-app designer, a conventional shopping section allows customers to order new or replacement tiles and supplies in a more traditional manner.

The development of the flor.com iPad application resulted in several positive outcomes:

Custom Carpet Design Experience: The in-app designer empowers customers to experiment and create unique carpet designs, catering to individual preferences.

Diverse Ordering Options: Customers can choose from a gallery of predesigned carpets for quick selection or opt for a more traditional shopping experience in the general shopping section.

Intuitive User Interface: The application offers an intuitive and user-friendly interface, ensuring a seamless and enjoyable shopping experience on the iPad.

Versatile Carpet Configurations: The flexibility to create carpets of any shape and size, using various tile combinations, enhances the versatility of available options.

Innovative Shopping Approach: The application showcases an innovative approach to shopping for rug and carpet tiles, aligning with flor.com's commitment to innovation.

In conclusion, the flor.com iPad application exemplifies a successful blend of innovation and functionality in the rug and carpet shopping experience. The in-app designer, predesigned carpet gallery, and traditional shopping section together offer customers a versatile and enjoyable way to explore and order custom carpet designs through their iPads.

This case study outlines the evolution of a business analytics ETL (Extract, Transform, Load) pipeline architecture. It details the challenges faced with Airflow and expensive AWS EC2 instances running Python's Spark, leading to a transition to AWS Step Functions/Lambdas. The study then explores the implementation of a novel approach involving separate Git repositories per pipeline, AWS CDK for automated deployments, and the development of supporting tools like a Slack bot and an internal admin web UI.

The initial ETL pipelines, built with Airflow and Python's Spark on AWS EC2 instances, proved to be slow and incurred high operational costs. Transitioning to AWS Step Functions/Lambdas introduced management complexities and demanded significant effort to handle numerous steps effectively.

The project evolved through multiple phases, introducing a new architecture and improving operational efficiency:

Git Repositories and AWS CDK: Each pipeline now has a dedicated Git repository with AWS CDK code for automated deployment, fostering better version control and management.

Code Reviews and QA Process: A robust process for code reviews, quality assurance, and releases was established to enhance project governance.

Technology Transition: Existing projects were re-written using the new approach, leveraging AWS Step Functions and Lambdas.

Slack-Bot Integration: A Slack bot was implemented, enabling manual pipeline execution through simple commands in the chat, simplifying the process for the team.

Admin Web-UI Solution: An internal web-based administration tool was developed to monitor pipelines and manage configurations efficiently.

The transformation led to numerous positive outcomes:

Enhanced Efficiency: The new architecture significantly improved the efficiency of ETL pipelines, reducing operational costs and runtime.

Version Control: Separate Git repositories per pipeline allowed for better version control, code organization, and simplified collaboration.

Improved Governance: Introduction of code reviews, QA processes, and a structured release approach enhanced project governance.

User-Friendly Operations: The Slack bot simplified manual pipeline execution, making it more accessible and user-friendly for the team.

Centralized Administration: The internal admin web UI provided a centralized platform for monitoring and configuring pipelines, streamlining administrative tasks.

In conclusion, the business analytics ETL pipeline transformation project showcases the successful evolution from traditional Airflow and EC2 instances to a modern, efficient, and user-friendly architecture. The adoption of separate Git repositories, automated deployments, and supporting tools significantly enhanced operational workflows, fostering a more streamlined and manageable ETL process.

This case study explores the development of a robust and customizable web platform tailored for US public utilities. The platform aimed to streamline workload distribution, management, and control for various tasks, such as water meter reading, asset assessment, pipe replacement, and garbage removal. The resulting system needed to be reliable, always up-to-date, flexible, and support multi-platform usage. The study details the architecture, modules, and achievements of the platform, which has garnered acclaim and was eventually acquired by Xylem, a market leader in innovative water solutions.

The challenge was to create a web platform that caters to the diverse needs of US public utilities, ensuring reliability, constant synchronization for offline devices, flexibility for different clients, and support for both desktop and mobile usage.

The team devised a comprehensive solution with the following key components:

Architecture:

Frontend: Three single-page JavaScript/AngularJS web applications for administrators, dispatchers, and field workers.

Backend: Written in C# with an MS SQL database for data storage.

Communication: Frontend and backend communicate via RabbitMQ message-broker, ensuring real-time updates and synchronization.

Configurable Modules:

Work & Asset Management

User Management

User Group Management

Routes Building

UI Customization

State Management

Automatic & Manual Work Scheduling

Preventative Maintenance Scheduling

Inventory Management

Shift Management

Measurements Collection

Errors Handling

Shortcuts Definition

Messaging

Results:

Frontend applications for administrators, dispatchers, and field workers.

Backend in C# with an MS SQL database.

RabbitMQ for real-time communication and synchronization.

Configurable modules for diverse utility management needs.

Acquisition by Xylem in 2019, highlighting the platform's success and industry recognition.

The outcomes of this transformative project include:

Comprehensive Platform: A versatile web platform catering to utilities management with frontend applications, a robust backend, and configurable modules.

Successful Implementation: Adoption by more than 10 satisfied customers, including major players like Denver Water, Badger Meter, National Meter, Western Disposal, etc.

Market Recognition: Acquisition by Xylem, a leading company in innovative water solutions, in 2019, indicating the platform's industry impact and success.

In conclusion, this case study showcases the team's ability to address complex challenges in utilities management through innovative technology solutions. The success of the platform, recognized by major industry players, underscores its effectiveness in meeting the diverse needs of US public utilities.

This case study tracks the evolution of preinstalled automotive navigation software, driven by the dynamic requirements of modern in-car infotainment systems. Focused on delivering high-resolution, smooth operation, and feature-rich experiences, the software addresses user expectations and business needs.

Advancements in hardware created new expectations for in-car infotainment systems, especially navigation software. Users demanded high-resolution displays, smooth operation, clear presentations, 3D maps, and recognizable landmarks. Meanwhile, business requirements evolved to include lane-level guidance, 3D guidance for multi-level junctions, electric vehicle range tracking, and more. These features needed to be delivered across various hardware platforms and in diverse visual designs matching the styles of different car manufacturers.

The project addressed these challenges through a multi-faceted solution:

In-House 3D Engine Adaptation: An in-house 3D engine was adapted to meet the specific requirements of map presentation in the automotive navigation context.

Feature Implementation: Essential features were implemented, including the display and prioritization of road names, clustering of point of interest icons, showing road and direction signs, and general optimization for limited hardware resources.

Hardware Compatibility: The software was engineered to be compatible with a variety of hardware platforms, ensuring a seamless experience across different in-car infotainment systems.

Visual Design Customization: Visual designs were tailored to match the unique styles of various car manufacturers, ensuring a cohesive and branded user experience.

The project yielded several positive outcomes:

Enhanced User Experience: The software met and exceeded user expectations by delivering high-resolution, smooth, and feature-rich navigation experiences.

Business Requirements Addressed: Lane-level guidance, 3D guidance for multi-level junctions, electric vehicle range tracking, and other business requirements were successfully implemented.

Wide Adoption: The navigation software is now available in numerous cars produced by the Volkswagen group, including VW, Audi, Porsche, and in other vehicles from manufacturers like Honda.

Versatile Hardware Compatibility: The software seamlessly runs on various hardware platforms, catering to the diverse in-car infotainment systems in the market.

Brand Consistency: Visual designs were customized to match the styles of different car manufacturers, ensuring a consistent and branded experience for users.

In conclusion, the evolution of preinstalled automotive navigation software demonstrates the successful adaptation of technology to meet evolving user and business requirements. The resulting software is now a key feature in a range of vehicles, providing a high-quality and versatile in-car navigation experience.