​
Core Components

At the heart of Stream AI’s innovation is its Data Registry, a comprehensive repository of curated, parsed, structured data primed for AI consumption. This Data Registry forms the foundation of a modular AI stack, empowering developers to selectively choose datasets for creating highly differentiated AI models. Envision it as a distilled version of the internet, meticulously optimized for AI training.

The Stream AI ecosystem consists of several key components working in harmony to create a robust, decentralized data provisioning network:

a) Edge Nodes (ENs): Smart devices running the Stream AI app, contributing their spare resources to the network and transforming raw web data into structured datasets. These nodes operate at the edge of the network, closest to the data sources.

b) Gateway Nodes (GNs): Servers with public IP addresses that distribute scraping tasks among Edge Nodes, ensuring efficient resource utilization and geographical diversity.

c) Validators: Nodes responsible for verifying the integrity and quality of collected data, maintaining the overall health of the network.

d) AI Scraper Agent: An intelligent component that enables Edge Nodes to transform raw web data into structured datasets suitable for AI model training and fine-tuning.

e) ZK Processor: A cryptographic system that creates proofs of metadata, documenting the origin of every dataset and ensuring transparency in the AI training process.

f) AI Developers and Customers: The end-users of the Stream AI protocol who pay for access to high-quality, structured datasets. This group includes AI researchers, companies developing AI products, and organizations seeking to leverage AI for various applications.

g) AI Data Agent: Receive and understand public data requests from AI Models/Applications, make judgement, choose the existing Data Formula or Generate a new Data Formula.

​
Decentralized Web Scraping: The Core Innovation

Stream AI’s decentralized web scraping system revolutionizes the way data is collected and processed for AI training and RAG building. By distributing the workload across a vast network of devices, Stream AI overcomes the limitations of traditional centralized scraping methods, offering unparalleled scalability, efficiency, and geographical diversity.

Key features of Stream AI’s decentralized web scraping include:

a) Mobile-First Approach: Prioritizing mobile traffic sharing to leverage the vast potential of smart devices and expand network coverage.

b) Advanced Protocols: Support for various protocols, including SOCKS, enabling data scraping from both websites and mobile applications.

c) Efficient Task Distribution: Dividing scraping tasks into smaller, manageable requests distributed among Edge Nodes, minimizing network requirements and ensuring smooth user experiences.

d) Intelligent Traffic Control: Employing a TCP/IP packet transfer approach for fine-grained control over network traffic, optimizing data flow and reducing the risk of detection or blocking by target websites.

e) Enhanced Scraping Capabilities: Dedicated AI Scraper Agent, transforms Edge Nodes into powerful data processing hubs by executing complex web research tasks in seconds, adapting to anti-bot measures, and simulating diverse locations.

​
Data Quality and Integrity

Stream AI places a strong emphasis on data quality and integrity, implementing several mechanisms to ensure the collected data is reliable, diverse, and ethically sourced:

a) Validator Network: A dedicated network of nodes that verify the quality and authenticity of collected data.

b) Reputation System: An AI-powered system that evaluates the performance and reliability of Edge Nodes, incentivizing high-quality contributions.

c) ZK Processor: Creating cryptographic proofs of metadata to document the origin and provenance of each dataset.

d) Ethical Data Collection: Focusing solely on publicly available web data to preserve user privacy and comply with data protection regulations.

IntroductionThe Stream AI Protocol: Technical Architecture