Market Insights Server
jashchawla8
README
Market Insights Server
A real-time commodity tracking system using Apache Spark, OpenAI GPT, and the MCP protocol to generate actionable market insights.
Features
- Real-time data collection from Reddit, News APIs, and Yahoo Finance
- Scalable processing using Apache Spark (PySpark 3.5.0)
- Natural language insights powered by GPT-4
- Configurable for any commodity market: energy, metals, agriculture, and more
- Built-in dynamic configuration generation and subreddit discovery
- Ready for deployment with error handling, retries, and async collection
Requirements
- Python 3.x
- PySpark 3.5.0
- Spark NLP 4.4.0
- aiohttp
- yfinance
- openai
- beautifulsoup4
Install dependencies:
pip install -r requirements.txt
Usage
python spark_market_insights_server.py --commodity "nickel"
Outputs:
- Cleaned text data from Reddit and news sources
- TF-IDF features
- GPT-4-powered insight report
- JSON export of insights
Architecture
-
Data Collection Layer
- Async scraping of Reddit and news articles
- Yahoo Finance for live price feeds
-
Processing Layer (Apache Spark)
- Tokenization → Stop words removal → TF-IDF vectorization
- Supports Spark NLP pipelines
-
AI Insight Layer
- Uses OpenAI GPT-4 to summarize and synthesize market narratives
-
Configuration Layer
- Automatically identifies relevant subreddits and keywords per commodity
🔍 Example Output
“Nickel prices rose sharply after Indonesia’s new export ban. Reddit sentiment is bullish, with posts anticipating supply constraints. Market data shows correlated uptick in EV-related equities like NIO and LI.”
🛠 Troubleshooting
| Problem | Fix |
|---|---|
| Spark stage stuck | Check memory settings, repartition input |
| API returns 429 | Add backoff/retry logic, rotate API keys |
| GPT returns empty | Use latest models, tune prompt |
Coming Soon
- Youtube videos to be analyzed as well for the commodities
- Public dashboard on Smithery.ai
Contributing
Have a new data source or insight model? PRs welcome!
License
MIT License
Recommended Servers
Crypto Price & Market Analysis MCP Server
A Model Context Protocol (MCP) server that provides comprehensive cryptocurrency analysis using the CoinCap API. This server offers real-time price data, market analysis, and historical trends through an easy-to-use interface.
MCP PubMed Search
Server to search PubMed (PubMed is a free, online database that allows users to search for biomedical and life sciences literature). I have created on a day MCP came out but was on vacation, I saw someone post similar server in your DB, but figured to post mine.
dbt Semantic Layer MCP Server
A server that enables querying the dbt Semantic Layer through natural language conversations with Claude Desktop and other AI assistants, allowing users to discover metrics, create queries, analyze data, and visualize results.
mixpanel
Connect to your Mixpanel data. Query events, retention, and funnel data from Mixpanel analytics.
Sequential Thinking MCP Server
This server facilitates structured problem-solving by breaking down complex issues into sequential steps, supporting revisions, and enabling multiple solution paths through full MCP integration.
Nefino MCP Server
Provides large language models with access to news and information about renewable energy projects in Germany, allowing filtering by location, topic (solar, wind, hydrogen), and date range.
Vectorize
Vectorize MCP server for advanced retrieval, Private Deep Research, Anything-to-Markdown file extraction and text chunking.
Mathematica Documentation MCP server
A server that provides access to Mathematica documentation through FastMCP, enabling users to retrieve function documentation and list package symbols from Wolfram Mathematica.
kb-mcp-server
An MCP server aimed to be portable, local, easy and convenient to support semantic/graph based retrieval of txtai "all in one" embeddings database. Any txtai embeddings db in tar.gz form can be loaded
Research MCP Server
The server functions as an MCP server to interact with Notion for retrieving and creating survey data, integrating with the Claude Desktop Client for conducting and reviewing surveys.