# Jira Projects Sync

**Automated system to sync Jira tickets from egovhealthcare and openhealthcare organizations with daily consolidated reports**

## Overview

The `jira_projects` repository provides an automated pipeline for syncing Jira tickets from **egovhealthcare** and **openhealthcare** organizations. It fetches raw ticket data with descriptions, attachments, assignees, and comments, then processes and classifies the data into beautiful daily HTML reports showing what changed, what's assigned to you, what got completed, newly created items, and per-person daily work.

## What It Does

### Core Functionality

1. **Fetches Tickets from Both Jira Organizations**
   - Connects to egovhealthcare and openhealthcare Jira instances via REST API
   - Fetches all tickets from all accessible projects
   - Downloads attachments (images, PDFs, docs)
   - Captures full ticket data: description, comments, assignee, status, priority, labels, etc.
   - Maintains ticket history for change detection
   - Stores raw JSON with full metadata

2. **Classifies & Processes Data**
   - **Recently Status Updated**: Tickets with status changes today/yesterday
   - **Assigned to Me**: All tickets currently assigned to you
   - **Completed**: Tickets marked as Done/Closed/Resolved recently
   - **Newly Created**: Tickets created today/yesterday
   - **Per Person Work**: What each team member worked on today

3. **Generates Beautiful HTML Reports**
   - Self-contained HTML reports per date (like Projects app)
   - Tabbed interface: Overview, My Tickets, Status Changes, Completed, New, Team Activity, Needs Attention
   - Status badges, priority indicators, assignee avatars
   - Responsive design with gradient header
   - Embedded charts and metrics

4. **Interactive Dashboard**
   - Calendar view showing report availability
   - Date picker with navigation
   - iframe-based report viewer
   - Manifest.json for report indexing
   - Responsive design (desktop/tablet/mobile)

5. **Automated Daily Sync**
   - GitHub Actions workflow runs daily at 8:00 AM IST (2:30 AM UTC)
   - Only commits if changes detected
   - Fault-tolerant (one org failure doesn't stop the other)

## Key Features

### 1. Multi-Organization Support
- Sync from egovhealthcare and openhealthcare simultaneously
- Environment variable configuration
- Isolated data directories per organization
- Independent authentication per org

### 2. Comprehensive Data Capture
- Ticket metadata: key, summary, description, status, priority, assignee, reporter, labels, components
- Comments: author, body, timestamp
- Attachments: filenames, URLs, metadata
- Change history: status transitions, assignee changes
- Custom fields: sprint, story points, epic link, etc.

### 3. Smart Classification
- **Status Changes**: Detects tickets that changed status in last 24h
- **My Tickets**: Filters tickets assigned to jagan.kumar@egovernments.org
- **Completed**: Tracks Done/Closed/Resolved tickets
- **New Tickets**: Identifies tickets created in last 24h
- **Team Activity**: Groups work by person and shows daily contributions

### 4. Beautiful Reports
- Based on Projects app design system
- Color scheme: Purple/blue gradient (#667eea, #764ba2)
- Status badges: color-coded (Done=green, In Progress=blue, Todo=gray)
- Priority indicators: High=red, Medium=yellow, Low=green
- Assignee avatars with initials
- Metrics cards with counts and percentages

## Directory Structure

```
jira_projects/
├── raw_jira/                      # Raw JSON from Jira
│   ├── egovhealthcare/            # eGov organization
│   │   ├── projects/              # Project metadata
│   │   ├── issues/                # Issue details
│   │   ├── attachments/           # Downloaded files
│   │   └── _metadata.json         # Change tracking + timestamps
│   └── openhealthcare/            # OpenHealthcare organization
│       ├── projects/
│       ├── issues/
│       ├── attachments/
│       └── _metadata.json
├── processed_reports/             # Generated HTML reports
│   ├── daily/
│   │   └── YYYY-MM-DD_runN_RUNID.html
│   └── weekly/                    # Future: weekly rollups
│       └── YYYY-MM-DD_runN_RUNID.html
├── dashboard/                     # Frontend UI
│   ├── index.html                 # Main dashboard
│   ├── app.js                     # Report loading logic
│   ├── styles.css                 # Dashboard styling
│   └── manifest.json              # Report index
├── scripts/                       # Python scripts
│   ├── fetch_jira.py              # Jira API fetcher
│   ├── process_and_generate.py   # Classifier + HTML generator
│   └── update_manifest.py         # Manifest indexer
├── .github/workflows/
│   └── sync-jira.yml              # Daily automation
├── requirements.txt               # Python dependencies
├── .env.example                   # Environment variable template
└── README.md                      # This file
```

## Technology Stack

| Component | Technology |
|-----------|-----------|
| **Language** | Python 3.11+ |
| **Jira API** | jira (Python library) |
| **HTML Generation** | Jinja2 templates |
| **Data Processing** | JSON, datetime |
| **Frontend** | Vanilla JavaScript, HTML5, CSS3 |
| **CI/CD** | GitHub Actions |
| **Authentication** | Jira API tokens |

## Quick Start

### 1. Install Dependencies

```bash
cd jira_projects
pip install -r requirements.txt
```

### 2. Configure Environment Variables

```bash
# Copy example file
cp .env.example .env

# Edit .env with your credentials
vim .env
```

Required variables:
```bash
# eGov Healthcare Organization
JIRA_EGOVHEALTHCARE_URL="https://egovhealthcare.atlassian.net"
JIRA_EGOVHEALTHCARE_USERNAME="jagan.kumar@egovernments.org"
JIRA_EGOVHEALTHCARE_API_TOKEN="your-api-token-here"

# OpenHealthcare Organization
JIRA_OPENHEALTHCARE_URL="https://openhealthcarenetwork.atlassian.net"
JIRA_OPENHEALTHCARE_USERNAME="jagan.kumar@egovernments.org"
JIRA_OPENHEALTHCARE_API_TOKEN="your-api-token-here"

# Your username for "Assigned to Me" filtering
MY_JIRA_USERNAME="jagan.kumar"

# Timezone
TZ="Asia/Kolkata"
```

### 3. Run Manual Sync

```bash
# Fetch raw Jira data from both organizations
python scripts/fetch_jira.py

# Process and generate reports
python scripts/process_and_generate.py

# Update manifest for dashboard
python scripts/update_manifest.py
```

### 4. View Dashboard

Open `dashboard/index.html` in your browser, or serve locally:

```bash
cd dashboard
python -m http.server 8000
# Visit http://localhost:8000
```

## Configuration

### Environment Variables

| Variable | Description | Example |
|----------|-------------|---------|
| `JIRA_EGOVHEALTHCARE_URL` | eGov Jira instance URL | `https://egovhealthcare.atlassian.net` |
| `JIRA_EGOVHEALTHCARE_USERNAME` | Email for eGov authentication | `jagan.kumar@egovernments.org` |
| `JIRA_EGOVHEALTHCARE_API_TOKEN` | API token for eGov | `ATATT3xFfGF0...` |
| `JIRA_OPENHEALTHCARE_URL` | OpenHealthcare Jira instance URL | `https://openhealthcarenetwork.atlassian.net` |
| `JIRA_OPENHEALTHCARE_USERNAME` | Email for OpenHealthcare auth | `jagan.kumar@egovernments.org` |
| `JIRA_OPENHEALTHCARE_API_TOKEN` | API token for OpenHealthcare | `ATATT3xFfGF0...` |
| `MY_JIRA_USERNAME` | Your Jira username (for filtering) | `jagan.kumar` |
| `TZ` | Timezone (default: Asia/Kolkata) | `Asia/Kolkata` |

### Getting Jira API Tokens

1. Go to https://id.atlassian.com/manage-profile/security/api-tokens
2. Click "Create API token"
3. Give it a label (e.g., "Jira Sync - eGov")
4. Copy the token immediately (you can't view it again)
5. Repeat for second organization if needed
6. Add to `.env` file

## Report Structure

Each daily report contains **7 tabs**:

### 1. Overview Tab
- **Metrics Cards**:
  - Total tickets tracked (both orgs combined)
  - Status changed today
  - Assigned to me
  - Completed today
  - Newly created today
- **Organization breakdown**: egovhealthcare vs openhealthcare ticket counts
- **Priority distribution**: High/Medium/Low counts

### 2. My Tickets Tab
- All tickets currently assigned to **jagan.kumar**
- Grouped by status (Done, In Progress, Todo, etc.)
- Shows: Key, Summary, Priority, Status, Comments count, Organization
- Quick links to Jira

### 3. Status Changes Tab
- Tickets that changed status in last 24 hours
- Grouped by status transition:
  - Todo → In Progress
  - In Progress → In Review
  - In Review → Done
  - etc.
- Shows: Key, Summary, Assignee, Old Status → New Status, Timestamp

### 4. Completed Tab
- Tickets marked Done/Closed/Resolved today/yesterday
- Shows who completed them
- Time to completion metric
- Completion velocity graph

### 5. New Tickets Tab
- Tickets created in last 24 hours
- Shows: Creator, Assignee, Priority, Labels, Organization
- Unassigned tickets highlighted

### 6. Team Activity Tab
- Per-person work summary
- Shows what each team member worked on today
- Grouped by person → ticket list with status changes, comments, updates
- Activity heatmap

### 7. Needs Attention Tab
- **Stale tickets**: No updates in 7+ days
- **High priority not in progress**: Urgent tickets in Todo/Backlog
- **Blocked tickets**: Marked as blocked/impediment
- **Missing assignee**: Unassigned tickets
- **Overdue**: Past due date

## Data Processing Logic

### Classification Rules

**Recently Status Updated**:
```python
updated_within_24h = ticket['updated'] >= (now - 24 hours)
status_changed = 'status' in ticket['changelog'][-1]['items']
```

**Assigned to Me**:
```python
assigned_to_me = (
    ticket['fields']['assignee'] and
    ticket['fields']['assignee']['name'] == 'jagan.kumar'
)
```

**Completed**:
```python
completed = ticket['fields']['status']['name'] in ['Done', 'Closed', 'Resolved']
completed_recently = completed and updated_within_24h
```

**Newly Created**:
```python
newly_created = ticket['fields']['created'] >= (now - 24 hours)
```

**Per Person Work**:
```python
person_activity = {
    person: [
        ticket for ticket in tickets
        if worked_on_today(ticket, person)
    ]
    for person in all_assignees
}

def worked_on_today(ticket, person):
    # Check if person commented, updated status, or is assignee with updates today
    return (
        any(comment['author'] == person and is_today(comment['created'])
            for comment in ticket['comments']) or
        (ticket['assignee'] == person and ticket['updated_today'])
    )
```

### Change Detection

Tracks changes by comparing current state to previous sync:
- Status transitions (with from/to values)
- Assignee changes
- Comment additions (new comments since last sync)
- Description updates
- Attachment additions

Stored in `_metadata.json` per organization:
```json
{
  "last_sync": "2026-09-10T08:00:00+05:30",
  "tickets_fetched": 1234,
  "changes_detected": {
    "status_changes": 45,
    "new_comments": 78,
    "new_tickets": 12
  },
  "ticket_checksums": {
    "CARE-123": "md5_hash_of_content"
  }
}
```

## Dashboard Features

### Calendar View
- Month-by-month calendar (Jan 2026 - Dec 2026)
- Highlights dates with available reports (green dot)
- Click date to load that day's report
- Today highlighted with blue border

### Navigation
- **Previous/Next day** buttons (< >)
- **Date picker** dropdown with all available dates
- **Jump to today** button
- **Refresh** button to reload manifest

### Report Display
- iframe-based embedding (full report rendered inside)
- Responsive height adjustment
- Smooth loading transitions
- Fallback message if report not found

### Manifest System
- `manifest.json` indexes all reports by date
- Enables fast date lookup (O(1))
- Powers calendar highlighting
- Format:
  ```json
  {
    "generated": "2026-09-10T08:00:00Z",
    "reports": {
      "daily": [
        {
          "date": "2026-09-10_run1_12345678",
          "file": "processed_reports/daily/2026-09-10_run1_12345678.html",
          "label": "Wed, Sep 10, 2026",
          "stats": {
            "total": 156,
            "my_tickets": 12,
            "completed": 8,
            "new": 5
          }
        }
      ]
    }
  }
  ```

## GitHub Actions Automation

Workflow runs daily at **8:00 AM IST (2:30 AM UTC)** Mon-Fri:

### Workflow Steps
1. Checkout repository
2. Set up Python 3.11
3. Install dependencies from requirements.txt
4. Load environment secrets
5. Run `fetch_jira.py` (fetches from egovhealthcare and openhealthcare)
6. Run `process_and_generate.py` (generates HTML report)
7. Run `update_manifest.py` (updates manifest.json)
8. Check for changes (git diff)
9. Commit changes with message: `chore: sync jira data for YYYY-MM-DD`
10. Push to repository

### Manual Trigger

You can manually trigger the sync anytime:
1. Go to Actions tab on GitHub
2. Select "Sync Jira Daily"
3. Click "Run workflow" dropdown
4. Select branch (usually `main`)
5. Click green "Run workflow" button

### Required GitHub Secrets

Navigate to Settings → Secrets and variables → Actions, then add:

| Secret Name | Value |
|-------------|-------|
| `JIRA_EGOVHEALTHCARE_URL` | `https://egovhealthcare.atlassian.net` |
| `JIRA_EGOVHEALTHCARE_USERNAME` | `jagan.kumar@egovernments.org` |
| `JIRA_EGOVHEALTHCARE_API_TOKEN` | Your eGov API token |
| `JIRA_OPENHEALTHCARE_URL` | `https://openhealthcarenetwork.atlassian.net` |
| `JIRA_OPENHEALTHCARE_USERNAME` | `jagan.kumar@egovernments.org` |
| `JIRA_OPENHEALTHCARE_API_TOKEN` | Your OpenHealthcare API token |
| `MY_JIRA_USERNAME` | `jagan.kumar` |

## Dashboard Deployment

Deploy your interactive dashboard to view reports from anywhere.

### Option 1: Cloudflare Pages (Recommended)

**Fast, global CDN deployment with auto-updates**

```bash
# Quick Start: See CLOUDFLARE_QUICK_START.md
```

**Setup (5 minutes):**
1. Go to [Cloudflare Pages](https://dash.cloudflare.com/)
2. Connect your GitHub repository
3. Set **Build output directory** to: `dashboard`
4. Deploy!

**Result:** `https://jira-projects-xxx.pages.dev`

**Benefits:**
- ⚡ **Fastest**: Global CDN (275+ cities)
- 🔄 **Auto-deploy**: Updates on every git commit
- 📊 **Analytics**: Built-in web analytics
- 🆓 **Free**: Unlimited bandwidth
- 🔒 **Secure**: Auto SSL, DDoS protection

**Full guide:** [CLOUDFLARE_PAGES_SETUP.md](./CLOUDFLARE_PAGES_SETUP.md)

### Option 2: GitHub Pages

**Simple GitHub-hosted deployment**

**Setup:**
1. Go to **Settings** → **Pages**
2. Source: **Deploy from branch**
3. Branch: **main** / Folder: **/ (root)**
4. Save

**Result:** `https://yourusername.github.io/jira_projects/dashboard/`

**Note:** Slower than Cloudflare, no built-in analytics

### Option 3: Local Server

**For testing or offline use**

```bash
cd dashboard
python -m http.server 8000
```

Visit: `http://localhost:8000`

### Which Option to Choose?

| Feature | Cloudflare Pages | GitHub Pages | Local |
|---------|-----------------|--------------|-------|
| **Speed** | ⚡⚡⚡ Excellent | 🟢 Good | 🟢 Good |
| **Setup** | 5 min | 2 min | 30 sec |
| **Auto-update** | ✅ Yes | ✅ Yes | ❌ No |
| **Analytics** | ✅ Yes | ❌ No | ❌ No |
| **Custom domain** | ✅ Easy | ✅ Yes | ❌ No |
| **Best for** | Production | Simple hosting | Development |

**Recommendation:** Use **Cloudflare Pages** for production deployment.

## Architecture Principles

### 1. Separation of Concerns
- **Fetching** (`fetch_jira.py`): API calls, downloads, raw storage
- **Processing** (`process_and_generate.py`): Classification, HTML generation
- **Indexing** (`update_manifest.py`): Manifest updates

### 2. Data Isolation
- Each organization gets separate directories
- No data mixing between egovhealthcare and openhealthcare
- Independent metadata tracking

### 3. Fault Tolerance
- One org failure doesn't stop the other
- Graceful error handling with detailed logging
- Partial updates are committed
- Retry logic with exponential backoff

### 4. Performance Optimization
- Incremental fetching (only changed tickets via JQL updated >= lastSyncTime)
- Attachment caching (skip if size unchanged)
- Efficient pagination for large projects
- Parallel fetching from both orgs

### 5. Idempotency
- Multiple runs produce same output for unchanged data
- Change detection via MD5 checksums
- Safe to run repeatedly without duplication

## Comparison with Projects App

| Feature | Projects App | Jira Projects Sync |
|---------|--------------|-------------------|
| **Data Source** | GitHub Projects GraphQL | Jira REST API (2 orgs) |
| **Fetching** | GraphQL query | REST API with JQL pagination |
| **Classification** | Epic-based + status | Status change, assignee, completion, per-person |
| **Team View** | Epic overview | Per-person activity |
| **Dashboard** | Calendar + tabs + iframe | Calendar + tabs + iframe (identical design) |
| **Styling** | Purple gradient (#667eea, #764ba2) | Same purple gradient |
| **Reports** | Self-contained HTML | Self-contained HTML (same structure) |
| **Automation** | Daily via GitHub Actions | Daily via GitHub Actions |
| **Timezone** | Asia/Kolkata | Asia/Kolkata |

Both systems share the same UI/UX design language for consistency.

## Security Considerations

### 1. API Token Management
- ✅ Use API tokens (not passwords)
- ✅ Store in GitHub Secrets (never commit to .env)
- ✅ Rotate tokens every 90 days
- ✅ Use read-only access if possible
- ⚠️ Never share tokens via email/Slack

### 2. Data Sensitivity
- ⚠️ Assume all Jira content is **sensitive**
- ✅ Use **private repository** only
- ⚠️ Attachments may contain **PHI/PII**
- ✅ Review .gitignore to exclude local .env files
- ✅ Enable branch protection rules

### 3. Access Control
- Each org uses separate credentials
- Principle of least privilege (read-only tokens)
- Audit who has access to GitHub Secrets
- Monitor Actions workflow runs for anomalies

### 4. Data Retention
- Raw data kept for 30 days (rolling window)
- Processed reports kept indefinitely
- Attachments kept for 30 days
- Configure in `_metadata.json` retention policy

## Troubleshooting

### Issue: "No organizations configured"

**Cause**: Environment variables not set

**Fix**:
```bash
# Check if variables exist
env | grep JIRA_

# Load .env file manually
export $(cat .env | xargs)

# Verify
echo $JIRA_EGOVHEALTHCARE_URL
```

### Issue: "Authentication failed for egovhealthcare"

**Cause**: Invalid API token or username

**Fix**:
1. Verify username matches Jira account email
2. Regenerate API token at https://id.atlassian.com/manage-profile/security/api-tokens
3. Update .env file or GitHub Secret
4. Retry: `python scripts/fetch_jira.py`

### Issue: "Rate limit exceeded"

**Cause**: Too many API requests (Jira Cloud limit: 10 req/sec)

**Fix**:
- Script already implements exponential backoff
- Check if multiple workflows running simultaneously
- Wait 60 seconds and retry
- Consider reducing fetch frequency

### Issue: "Dashboard showing 'No reports found'"

**Cause**: manifest.json not updated or empty

**Fix**:
```bash
# Regenerate manifest
python scripts/update_manifest.py

# Check if reports exist
ls -la processed_reports/daily/

# Refresh browser (Ctrl+Shift+R)
```

### Issue: "Report shows 0 tickets"

**Cause**: JQL query returned no results

**Fix**:
- Check Jira permissions (can you see tickets in Jira UI?)
- Verify project keys are correct
- Check `raw_jira/*/issues/` for fetched data
- Review fetch logs for errors

### Issue: "My Tickets tab is empty"

**Cause**: MY_JIRA_USERNAME doesn't match Jira assignee field

**Fix**:
- Check your Jira username (Profile → Account settings)
- Update MY_JIRA_USERNAME in .env
- Common mistake: using email instead of username
  - ❌ `jagan.kumar@egovernments.org`
  - ✅ `jagan.kumar`

## Performance Metrics

### Typical Sync Times

| Tickets (per org) | First Run | Daily Run (5% changed) |
|-------------------|-----------|----------------------|
| 100               | 2m        | 15s                  |
| 500               | 6m        | 30s                  |
| 1000              | 12m       | 45s                  |
| 5000              | 1h        | 3m                   |

**Assumptions**:
- Jira Cloud standard tier
- Average 8 comments per ticket
- 15% of tickets have attachments
- Network latency ~100ms to atlassian.net
- Running from GitHub Actions (US East servers)

### API Rate Limits

**Jira Cloud**: 10 requests/second per IP

**Full fetch**: 1 request per ticket = high usage
**Incremental fetch**: JQL query for changed tickets only = low usage

**Example** (egovhealthcare: 500 tickets, 25 changed):
- Full: ~500 API calls
- Incremental: 1 JQL query + 25 ticket fetches = ~26 calls
- **95% reduction in API usage**

## Related Repositories

- **[confluence_docs](../confluence_docs/)** - Confluence sync (same architecture)
- **[Projects](../Projects/)** - GitHub Projects dashboard (UI reference)
- **[care-learnings](../care-learnings/)** - Knowledge base

## Future Enhancements

- [ ] Weekly rollup reports (summary of week's activity)
- [ ] Sprint velocity tracking (story points over time)
- [ ] Burndown charts (remaining work vs days)
- [ ] Custom field extraction (sprints, story points, epics)
- [ ] Email notifications for high-priority items assigned to you
- [ ] Slack/Discord integration (post daily summary)
- [ ] Advanced filtering (by label, component, epic, sprint)
- [ ] Search functionality in dashboard (Ctrl+K command palette)
- [ ] Export to CSV/Excel
- [ ] Integration with care_fe task system
- [ ] Dependency graph visualization
- [ ] Time tracking analytics

## Contributing

When adding features:
1. Maintain multi-org isolation (egovhealthcare and openhealthcare)
2. Preserve fault tolerance (one org failure shouldn't stop the other)
3. Add CLI flags for new behavior (`--full-sync`, `--org=egovhealthcare`)
4. Update README and inline documentation
5. Test with both organizations
6. Add unit tests for classification logic
7. Follow Projects app UI patterns for consistency

## Resources

- [Jira REST API Documentation](https://developer.atlassian.com/cloud/jira/platform/rest/v3/intro/)
- [Jira Python Library](https://jira.readthedocs.io/en/latest/)
- [JQL Reference](https://support.atlassian.com/jira-service-management-cloud/docs/use-advanced-search-with-jira-query-language-jql/)
- [Jira API Rate Limits](https://developer.atlassian.com/cloud/jira/platform/rate-limiting/)
- [Projects App Reference](../Projects/README.md)

---

**Organizations**: egovhealthcare, openhealthcare
**Maintainer**: Jagan Kumar (jagan.kumar@egovernments.org)
**Version**: 1.0.0
**Last Updated**: 2026-09-10
**Status**: Production Ready