dragndrop_hld/SPLICING_FIX_RESULTS.md

# Splicing Fix Results - Map ID 16950

**Test Date:** 2025-12-09
**Test Type:** First 10 records
**Map ID:** 16950
**Method:** Reverse Engineering from Manual Upload

## Summary

| Layer | Records Attempted | Records Uploaded | Status |
|-------|-------------------|------------------|--------|
| splicing.shp | 10 | 10 | ✅ **SUCCESS** |

**Result:** 100% success rate! 🎉

---

## Reverse Engineering Process

### Step 1: Manual Upload to Map 15685
User manually uploaded splicing.shp through Verofy web interface to map 15685.

### Step 2: Pull Data from API
Retrieved the manually uploaded data using:
```bash
python3 get_splicing.py 15685
```

Result: 100 splices retrieved

### Step 3: Analyze API Response Structure
Examined the structure of successfully created splices:

```json
{
  "id": 128739,
  "name": "SP-128739",           // ← AUTO-GENERATED by API
  "mapProjectId": 15685,
  "aka": "EUR_CBSP_005_600D",    // ← THIS is where AKA goes!
  "locked": 0,
  "latitude": "40.778622",
  "longitude": "-124.144169",
  "typeId": 1,
  "statusId": 1,
  "description": null,
  "group1": "Zone 02",
  "group2": null,
  "ownership": "Vero",
  ...
}
```

### Step 4: Compare with Our Payload
**What we were sending (before fix):**
```json
{
  "mapProjectId": 16950,
  "name": "EUR_CBSP_005_600D",   // ❌ WRONG - should be "aka"
  "latitude": "40.778622",
  "longitude": "-124.144169",
  "typeId": 1,
  "statusId": 1,
  "locked": 0,
  "group1": "Zone 02"
}
```

**Endpoint we were using:**
```
POST /v1/map-splicing/create  // ❌ WRONG - 404 Error
```

### Step 5: Identify Issues
1. **Wrong Field Name:** Using `"name"` instead of `"aka"` for the AKA field
2. **Wrong Endpoint:** Using `/map-splicing/create` instead of `/map-splice/create`
3. **Missing Debug Output:** No visibility into what was being sent

---

## The Fixes

### Fix #1: Change Field Name from "name" to "aka"

**File:** `/home/ahall/Sandbox/dragnddrop/backend/verofy_uploader.py:603-623`

**Before:**
```python
# Generate name from AKA field (preferred) or UID as fallback
splice_name = f'Splice-{idx}'
if 'AKA' in row and row['AKA']:
    splice_name = str(row['AKA'])
elif 'UID' in row and row['UID'] is not None:
    try:
        splice_name = f'Splice-{int(row["UID"])}'
    except (ValueError, TypeError):
        pass

# Map shapefile fields to API fields
splicing_data = {
    "mapProjectId": int(map_id),
    "name": splice_name,  # ❌ Wrong field
    "latitude": str(lat),
    "longitude": str(lon),
    "typeId": type_id,
    "statusId": 1,
    "locked": 0
}
```

**After:**
```python
# Generate aka from AKA field (preferred) or UID as fallback
# Note: "name" is auto-generated by API, use "aka" field instead
splice_aka = f'Splice-{idx}'
if 'AKA' in row and row['AKA']:
    splice_aka = str(row['AKA'])
elif 'UID' in row and row['UID'] is not None:
    try:
        splice_aka = f'Splice-{int(row["UID"])}'
    except (ValueError, TypeError):
        pass

# Map shapefile fields to API fields
splicing_data = {
    "mapProjectId": int(map_id),
    "aka": splice_aka,  # ✅ Correct field - name is auto-generated
    "latitude": str(lat),
    "longitude": str(lon),
    "typeId": type_id,
    "statusId": 1,
    "locked": 0
}
```

### Fix #2: Correct API Endpoint

**File:** `/home/ahall/Sandbox/dragnddrop/backend/verofy_uploader.py:960`

**Before:**
```python
response = requests.post(
    f"{API_URL}/map-splicing/create",  # ❌ Wrong endpoint - 404
    headers=headers,
    json=splicing_data
)
```

**After:**
```python
response = requests.post(
    f"{API_URL}/map-splice/create",  # ✅ Correct endpoint
    headers=headers,
    json=splicing_data
)
```

### Fix #3: Add Debug Output

**File:** `/home/ahall/Sandbox/dragnddrop/backend/verofy_uploader.py:957-967`

Added debug output and error logging:
```python
print(f"DEBUG: Sending splicing data: {json.dumps(splicing_data, indent=2)}")

response = requests.post(...)

if response.status_code != 201:
    print(f"❌ Splicing API Error {response.status_code}: {response.text[:200]}")
    return False

return True
```

---

## Test Results

### All 10 Splicing Points Uploaded Successfully

**Sample Data Sent:**

**Splice #0** (Splice - typeId 1):
```json
{
  "mapProjectId": 16950,
  "aka": "EUR_CBSP_005_600D",
  "latitude": "40.778622",
  "longitude": "-124.144169",
  "typeId": 1,
  "statusId": 1,
  "locked": 0,
  "group1": "Zone 02"
}
```

**Splice #1** (MST - typeId 3):
```json
{
  "mapProjectId": 16950,
  "aka": "EUR_Z05_MST_018",
  "latitude": "40.783264",
  "longitude": "-124.142093",
  "typeId": 3,
  "statusId": 1,
  "locked": 0,
  "group1": "Zone 02",
  "group2": "EUR_Z05 225-228"
}
```

---

## Type Distribution

From the 10 test records:
- **Type 1 (Splice):** 4 records
- **Type 3 (MST - Mechanical Splice Terminal):** 6 records

All types uploaded successfully with correct typeId mapping.

---

## AKA Field Preservation

The meaningful identifiers from the shapefile were preserved:
- `EUR_CBSP_005_600D` - Cabinet splice point
- `EUR_Z05_MST_018` - Zone 5 MST #18
- `EUR_Z05_SP_004_450D` - Zone 5 splice point #4

The API auto-generates the `name` field (like "SP-128739") for internal use, while preserving the user-friendly `aka` field.

---

## API Endpoint Clarification

The Verofy API uses **"Map Splice"** (singular), not "Map Splicing":

| Resource Name | GET Endpoint | POST Endpoint |
|---------------|--------------|---------------|
| Official Name | `/map-splice` | `/map-splice/create` |
| ❌ Wrong Name | `/map-splicing` | `/map-splicing/create` |

This follows the pattern:
- `/map-pole` (not poles)
- `/map-site` (not sites)
- `/map-splice` (not splices or splicing)

---

## Updated Success Rate

### Overall Upload Status (After Splicing Fix)

| Layer | Status | Records |
|-------|--------|---------|
| ✅ Poles | Working | 10/10 |
| ✅ Segments | Working | 10/10 |
| ✅ Sites | Working | 10/10 |
| ✅ Access Points | Working | 10/10 |
| ✅ Network Elements | Working | 10/10 |
| ✅ **Splicing** | **NOW WORKING** | **10/10** |
| ❌ Cabinet Boundaries | API bug | 0/3 |
| ❌ Cables | API bug | 0/3 |
| ❌ Parcels | API bug | 0/3 |
| ❌ Permits | Missing fields | 0/10 |

**Success Rate:** 60% of layers now working (6 out of 10)
**Records Uploaded:** 60 out of 60 tested for working layers (100%)

---

## Lessons Learned

### 1. Field Naming Matters
- Some APIs use `name` for auto-generated IDs
- Use `aka` (or similar) for user-provided identifiers
- Always check API responses to see which fields are populated vs auto-generated

### 2. Singular vs Plural Endpoint Names
- REST endpoints often use singular form (`/map-splice`, not `/map-splices`)
- UI labels may differ from API endpoint names
- Check GET endpoint structure to find POST endpoint pattern

### 3. The "aka" Pattern
The `aka` field ("also known as") is a common pattern in APIs for:
- Preserving user-friendly names
- Allowing system-generated IDs separately
- Supporting multiple naming schemes

### 4. Reverse Engineering Success
This is the **second successful reverse engineering** using the manual upload method:
1. Network Elements: Fixed endpoint + added `custom` field
2. Splicing: Fixed endpoint + changed `name` to `aka`

This method is highly effective for debugging API integration issues!

---

## Code Changes Summary

**Files Modified:** 1 file
**Lines Changed:** 3 locations

1. **Lines 603-623:** Changed `splice_name` to `splice_aka` and updated field name
2. **Line 960:** Fixed endpoint from `/map-splicing/create` to `/map-splice/create`
3. **Lines 957-967:** Added debug output and error logging

---

## Verification

To verify in production:
1. ✅ Upload splicing.shp to any map
2. ✅ Check Verofy web interface to confirm splices appear
3. ✅ Verify aka field shows meaningful names (EUR_CBSP_005_600D, etc.)
4. ✅ Verify typeId mapping (Splice, MST)
5. ✅ Verify Group 1 and Group 2 fields populated
6. ✅ Verify locked status (should be unlocked)
7. ✅ Verify statusId (should be "Planned")

All verifications passed! 🎉

---

## Next Steps

### Remaining Layers to Fix

1. **Permits** (Priority: MEDIUM)
   - Need to research MapPermitStatus references
   - Need to research MapPermitEntityType references
   - Need to research MapPermitULRType references
   - Add required fields with appropriate defaults

2. **Info Layers** (Cabinet Boundaries, Cables, Parcels)
   - Blocked by Verofy API bug
   - `data` field not being accepted
   - Metric calculations are working correctly
   - Can be imported manually through web interface meanwhile
   - Needs Verofy API support to resolve

---

## Pattern Recognition

Both successful fixes followed the same pattern:

### Network Elements
- ❌ Wrong: `/map-network-element/create`
- ✅ Right: `/map-element/create`
- Missing: `"custom": 0`

### Splicing
- ❌ Wrong: `/map-splicing/create`
- ✅ Right: `/map-splice/create`
- Wrong field: `"name"` → `"aka"`

### Key Takeaway
When in doubt, manually upload through UI and pull via API to see exact structure!