NYC Easy Enroll: Automating Public Benefit Access with AI-Powered Document Processing + Video

By /

Listen to this Post

Featured Image

Introduction:

The “Built for NYC” hackathon, hosted by The New York Public Library in partnership with Major League Hacking (MLH), challenged developers to leverage generative AI and “vibe coding” to address unique challenges facing New York City. The winning approach, NYC Easy Enroll, tackled a critical civic issue: the underutilization of public benefits due to complex, paper-heavy application processes. By building a secure, mobile-first application that automates document parsing and eligibility determination, the team demonstrated how AI can bridge the gap between citizens and essential city services.

Learning Objectives:

  • Understand how to architect a secure, privacy-preserving AI pipeline for processing sensitive Personally Identifiable Information (PII).
  • Learn to implement multimodal AI for structured data extraction from unstructured documents like IDs and tax forms.
  • Explore the integration of public datasets, such as NYC Open Data, with AI-driven logic to automate complex eligibility determinations.

You Should Know:

1. Multimodal AI for Structured Document Parsing

Traditional Optical Character Recognition (OCR) often fails to preserve the context and relationships between data fields on a document, leading to noisy, unstructured text. NYC Easy Enroll addressed this by utilizing Google AI Studio and the Gemini 1.5 Flash Vision model. This multimodal approach allows the AI to process the document as an image, using spatial awareness and context to extract key-value pairs directly into a structured JSON schema. This method is significantly more robust for handling varied document types like passports, driver’s licenses, and W-2 forms.

Step-by-step guide:

  1. Define a Pydantic Schema: Create a strict data model to define the expected output. This ensures the AI returns predictable, database-ready objects. 
    from pydantic import BaseModel</li>
</ol>

class UserProfile(BaseModel):
full_name: str
date_of_birth: str
ssn: str
annual_income: float
household_size: int
address: str

    2. Prepare the Craft a prompt that instructs the multimodal model to extract data and format it according to the defined JSON schema.

    "Extract the following information from this document image and return it as a JSON object matching the schema: full_name, date_of_birth, ssn, annual_income, household_size, address."

    3. Process the Image: Send the image and the prompt to the Gemini API.

    import google.generativeai as genai

model = genai.GenerativeModel('gemini-1.5-flash')
response = model.generate_content(["Extract user details as JSON...", image])

    4. Validate and Parse: Parse the API response and validate it against your Pydantic schema to ensure data integrity before storing or processing it further.

    user_data = UserProfile.parse_raw(response.text)

    2. Automating Eligibility Logic with NYC Open Data

    Translating the complex, legal eligibility requirements of programs like Fair Fares into code requires a reliable data source. The team used NYC Open Data, which provides access to numerous datasets, including 311 service requests, housing, and public safety information. The core logic involves pulling program requirements (e.g., income thresholds based on household size) from this data and comparing them against the user’s extracted profile.

    Step-by-step guide:

    1. Access the NYC Open Data API: The data is available via a Socrata API. You can query it directly or use helper libraries. 
      Example using curl to fetch 311 service request data
curl "https://data.cityofnewyork.us/resource/erm2-1we9.json?$limit=10"
    2. Fetch Program Requirements: Create a function to fetch the latest eligibility criteria for a specific program (e.g., Fair Fares). 
      import requests</li>
</ol>

def get_fair_fares_requirements():
 This is a simplified example. The actual API endpoint and query would be more complex.
response = requests.get("https://data.cityofnyc.us/api/programs/fair-fares")
return response.json()

      3. Implement Eligibility Check: Write a function that takes a `UserProfile` and the program’s requirements, then returns a boolean indicating eligibility.

      def is_eligible_for_fair_fares(user: UserProfile, requirements: dict):
income_limit = requirements['income_limit']  user.household_size
return user.annual_income <= income_limit

      4. Display Results: Use the calculated eligibility to update the user’s dashboard, perhaps with a “traffic-light” system (green for eligible, yellow for partial, red for ineligible).

      3. Building a Privacy-First Architecture for PII

      Handling sensitive data like Social Security Numbers (SSNs) demands a “privacy by design” approach. The application is architected so that raw document images are never stored permanently; they are processed in memory and immediately discarded. Only the essential, parsed data points needed for eligibility checks are stored.

      Step-by-step guide:

      1. Local Authentication: Implement strict local authentication, such as a phone PIN, before any data is processed or displayed.
      2. In-Memory Processing: When a user uploads a document, the file is read into memory, processed by the AI model, and then the file object is deleted. 
        Example using a temporary file that is deleted after processing
import tempfile
import os</li>
</ol>

with tempfile.NamedTemporaryFile(delete=True) as tmp_file:
tmp_file.write(uploaded_file_content)
 Process the file using the AI model
extracted_data = process_document(tmp_file.name)
 The file is automatically deleted when the 'with' block ends

        3. Encryption at Rest and in Transit: Ensure all stored data (even the parsed fields) is encrypted. Use HTTPS for all network communication.
        4. Data Minimization: Only store the absolute minimum data required for the eligibility check and application process. Avoid storing the full, raw text of documents.

        4. Leveraging “Vibe Coding” for Rapid Prototyping

        The team used “vibe coding” with GitHub Copilot to rapidly scaffold the frontend UI and backend infrastructure, dramatically accelerating development velocity. This approach, emphasized at the hackathon, involves using AI pair programmers to generate boilerplate code, allowing developers to focus on the unique and complex parts of the application.

        Step-by-step guide:

        1. Set Up an AI Coding Assistant: Install and configure GitHub Copilot in your IDE.
        2. Describe the Feature: Write a clear comment or prompt describing the functionality you need (e.g., “Create a React Native component for a file upload button”).
        3. Iterate and Refine: Review the AI-generated code, test it, and refine your prompts to get the desired output. This iterative process allows for rapid exploration and development of different parts of the application stack.

        5. Enforcing Structured Outputs from LLMs

        One of the key lessons learned was the importance of enforcing strict JSON schemas on LLM outputs. This ensures the AI returns predictable, database-ready objects instead of unstructured chatty text. This is critical for building reliable, automated pipelines where the output is consumed by other services.

        Step-by-step guide:

        1. Define Your Schema: Use a library like Pydantic to define the exact structure and data types you expect.
        2. Prompt with Schema: In your prompt to the LLM, clearly instruct it to return a JSON object that adheres to your schema.
        3. Parse and Validate: After receiving the response, parse the JSON and validate it against your schema. If validation fails, you can retry the request or flag the error for manual review.

        What Undercode Say:

        • AI as a Civic Bridge: NYC Easy Enroll shows that AI’s most powerful application isn’t just automation, but breaking down bureaucratic barriers that prevent citizens from accessing essential services.
        • Privacy is Paramount: Building trust with users, especially when handling sensitive data, is non-1egotiable. The “process and discard” model is a gold standard for privacy-preserving AI.

        The hackathon project successfully identified a real problem and built a functional, secure solution in a single weekend. The use of multimodal AI for document parsing, combined with a privacy-first architecture and the agility of “vibe coding,” provides a compelling blueprint for future civic tech projects. The team’s focus on empathy—designing a clear, step-by-step “traffic-light” system—transforms a stressful task into an empowering experience.

        Prediction:

        • +1 The success of NYC Easy Enroll will accelerate the adoption of AI-powered document processing in the public sector, leading to more efficient and accessible government services.
        • +1 Privacy-preserving architectures like “in-memory processing” will become a standard requirement for any application handling PII, setting a new benchmark for data security in civic tech.
        • -1 Scaling such a solution to handle city-wide demand will present significant infrastructure and data integration challenges that require careful planning and investment.

        ▶️ Related Video (86% Match):

        🎯Let’s Practice For Free:

        🎓 Live Courses & Certifications:

        Join Undercode Academy for Verified Certifications

        🚀 Request a Custom Project:

        Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
        [email protected]
        💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands

        IT/Security Reporter URL:

        Reported By: https://lnkd.in/p/ewFSU5UA – Hackers Feeds
        Extra Hub: Undercode MoN
        Basic Verification: Pass ✅

        🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

        💬 Whatsapp | 💬 Telegram

        📢 Follow UndercodeTesting & Stay Tuned:

        𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky

Related Posts: