Node.js Async Programming in SoftyFlow

This guide explains how to effectively use Node.js asynchronous programming patterns in SoftyFlow scripts, with focus on async/await syntax and best practices for writing robust, performant scripts.

Understanding Asynchronous Programming in SoftyFlow

SoftyFlow's JavaScript execution environment is built on Node.js, which means all I/O operations (database queries, API calls, file operations) are asynchronous. Understanding how to properly handle asynchronous operations is crucial for writing effective scripts.

The Promise-Based Architecture

All SoftyFlow SDK methods return Promises, which represent eventual completion (or failure) of asynchronous operations:

// ❌ Wrong - This won't work as expected
let users = SF.collection.find("users", {status: "active"}); // This returns a Promise object, not the data

// ✅ Correct - Using await to get the actual data
let users = await SF.collection.find("users", {status: "active"});

Async/Await Fundamentals

Basic Async/Await Pattern

The await keyword can only be used in functions marked with async. In SoftyFlow scripts, the execution context is automatically async, so you can use await directly:

// SoftyFlow script - automatically in async context
let user = await SF.user.getUserById("6042551d231e8c8ade19bc09");
let orders = await SF.collection.find("orders", {userId: user._id});

return {
  user: user,
  orderCount: orders.length,
  totalAmount: orders.reduce((sum, order) => sum + order.amount, 0)
};

Sequential vs Parallel Execution

Understanding when operations run sequentially vs in parallel is crucial for performance:

Sequential Execution (Operations Depend on Each Other)

// ✅ Good - When each operation depends on the previous one
let user = await SF.user.getUserById(userId);
let userGroup = await SF.user.getGroupById(user.groupId);
let permissions = await SF.collection.findOne("permissions", {groupId: userGroup._id});

return {
  user: user,
  group: userGroup,
  permissions: permissions
};

Parallel Execution (Independent Operations)

// ✅ Better - When operations are independent, run them in parallel
let [user, recentOrders, supportTickets] = await Promise.all([
  SF.user.getUserById(userId),
  SF.collection.find("orders", {userId: userId, createdAt: {$gte: lastMonth}}),
  SF.collection.find("tickets", {userId: userId, status: "open"})
]);

return {
  user: user,
  recentOrdersCount: recentOrders.length,
  openTicketsCount: supportTickets.length
};

Common SoftyFlow Async Patterns

1. Database Operations

Single Document Operations

// Find and update pattern
let existingRecord = await SF.collection.findOne("customers", {email: customerEmail});

if (existingRecord) {
  // Update existing customer
  await SF.collection.updateOne("customers", 
    {email: customerEmail}, 
    {lastLogin: new Date(), loginCount: existingRecord.loginCount + 1}
  );
} else {
  // Create new customer
  await SF.collection.insertOne("customers", {
    email: customerEmail,
    createdAt: new Date(),
    loginCount: 1
  });
}

Bulk Operations

// Efficient bulk processing
let usersToUpdate = await SF.collection.find("users", {status: "pending"});

// Prepare bulk operations
let bulkOps = usersToUpdate.map(user => ({
  updateOne: {
    filter: {_id: user._id},
    update: {$set: {status: "active", activatedAt: new Date()}}
  }
}));

// Execute all updates in one operation
if (bulkOps.length > 0) {
  await SF.collection.bulkWrite("users", bulkOps);
}

return {processed: usersToUpdate.length};

2. API Integration Patterns

// External API call with error handling
try {
  let response = await SF.utils.axios.get(`https://api.external-service.com/users/${userId}`, {
    headers: {
      'Authorization': `Bearer ${apiToken}`,
      'Content-Type': 'application/json'
    },
    timeout: 5000 // 5 second timeout
  });
  
  // Process the response
  let externalUser = response.data;
  
  // Update local database with external data
  await SF.collection.updateOne("users", 
    {_id: userId}, 
    {
      externalId: externalUser.id,
      lastSyncAt: new Date(),
      syncStatus: "success"
    }
  );
  
  return {success: true, externalId: externalUser.id};
  
} catch (error) {
  // Handle API errors gracefully
  console.error('External API call failed:', error.message);
  
  // Update sync status
  await SF.collection.updateOne("users", 
    {_id: userId}, 
    {syncStatus: "failed", lastSyncError: error.message}
  );
  
  return {success: false, error: error.message};
}

3. File Processing Patterns

// Generate PDF and save to collection
try {
  // Get template and data in parallel
  let [template, customerData] = await Promise.all([
    SF.file.getFileContentById(templateId),
    SF.collection.findOne("customers", {_id: customerId})
  ]);
  
  // Generate PDF
  let pdfFile = await SF.file.generatePDFFromTemplate(null, templateId, {
    variables: {
      customer: customerData,
      generatedAt: new Date().toLocaleDateString(),
      invoiceNumber: await SF.utils.nextValue("invoice", 6)
    },
    SF_pdfName: `invoice-${customerData.code}.pdf`
  });
  
  // Save reference in orders collection
  await SF.collection.updateOne("orders", 
    {customerId: customerId}, 
    {pdfFileId: pdfFile._id, pdfGeneratedAt: new Date()}
  );
  
  return {pdfGenerated: true, fileId: pdfFile._id};
  
} catch (error) {
  console.error('PDF generation failed:', error);
  return {pdfGenerated: false, error: error.message};
}

Best Practices for SoftyFlow Scripts

1. Error Handling

Always wrap potentially failing operations in try-catch blocks:

// ✅ Good - Robust error handling
async function processOrder(orderId) {
  try {
    let order = await SF.collection.findOne("orders", {_id: orderId});
    
    if (!order) {
      throw new Error(`Order ${orderId} not found`);
    }
    
    // Process order...
    let result = await processPayment(order);
    
    return {success: true, result: result};
    
  } catch (error) {
    console.error('Order processing failed:', error.message);
    
    // Log error for debugging
    await SF.collection.insertOne("error_logs", {
      operation: "processOrder",
      orderId: orderId,
      error: error.message,
      timestamp: new Date()
    });
    
    return {success: false, error: error.message};
  }
}

2. Performance Optimization

Use Parallel Operations When Possible

// ❌ Slow - Sequential operations
let user = await SF.user.getUserById(userId);
let orders = await SF.collection.find("orders", {userId: userId});
let preferences = await SF.collection.findOne("preferences", {userId: userId});

// ✅ Fast - Parallel operations
let [user, orders, preferences] = await Promise.all([
  SF.user.getUserById(userId),
  SF.collection.find("orders", {userId: userId}),
  SF.collection.findOne("preferences", {userId: userId})
]);

Minimize Database Queries

// ❌ Inefficient - Multiple queries
let results = [];
for (let userId of userIds) {
  let user = await SF.user.getUserById(userId);
  results.push(user);
}

// ✅ Efficient - Single aggregation query
let results = await SF.collection.aggregate("users", [
  {$match: {_id: {$in: userIds}}},
  {$project: {firstName: 1, lastName: 1, email: 1}}
]);

3. Variable Management

// ✅ Good - Clear variable organization
let processData = {
  // Input data validation
  customerId: customerId || null,
  orderDate: orderDate || new Date(),
  items: Array.isArray(items) ? items : [],
  
  // Calculated values
  subtotal: 0,
  tax: 0,
  total: 0,
  
  // Status tracking
  processed: false,
  errors: []
};

// Validate required data
if (!processData.customerId) {
  processData.errors.push("Customer ID is required");
}

if (processData.items.length === 0) {
  processData.errors.push("Order must contain at least one item");
}

if (processData.errors.length > 0) {
  return processData; // Return early with errors
}

// Process the order...

4. Conditional Processing

// ✅ Good - Clean conditional logic with async operations
let approvalRequired = false;
let approvers = [];

// Check if approval is needed based on amount
if (orderTotal > 10000) {
  approvalRequired = true;
  approvers = await SF.user.getUsersInGroupList(["managers", "directors"]);
} else if (orderTotal > 5000) {
  approvalRequired = true;
  approvers = await SF.user.getUsersinGroup("managers");
}

if (approvalRequired) {
  // Create approval task
  let approvalTask = await SF.instance.validateTask(currentTaskId, {
    requiresApproval: true,
    approvers: approvers.map(user => user.email),
    amount: orderTotal
  });
  
  return {
    status: "pending_approval",
    approvers: approvers.length,
    taskId: approvalTask.id
  };
} else {
  // Auto-approve small orders
  return {
    status: "approved",
    autoApproved: true
  };
}

Advanced Patterns

1. Retry Logic for External Services

async function callExternalServiceWithRetry(url, data, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      let response = await SF.utils.axios.post(url, data, {
        timeout: 10000
      });
      
      return response.data; // Success, return data
      
    } catch (error) {
      console.log(`Attempt ${attempt} failed:`, error.message);
      
      if (attempt === maxRetries) {
        throw new Error(`Failed after ${maxRetries} attempts: ${error.message}`);
      }
      
      // Wait before retry (exponential backoff)
      await new Promise(resolve => setTimeout(resolve, 1000 * attempt));
    }
  }
}

// Usage
try {
  let result = await callExternalServiceWithRetry("https://api.example.com/process", orderData);
  return {success: true, externalId: result.id};
} catch (error) {
  return {success: false, error: error.message};
}

2. Batch Processing with Progress Tracking

async function processBatchWithProgress(items, batchSize = 10) {
  let results = [];
  let processed = 0;
  let errors = [];
  
  // Process items in batches
  for (let i = 0; i < items.length; i += batchSize) {
    let batch = items.slice(i, i + batchSize);
    
    // Process batch in parallel
    let batchPromises = batch.map(async (item) => {
      try {
        let result = await SF.collection.insertOne("processed_items", {
          originalId: item.id,
          processedAt: new Date(),
          data: item
        });
        return {success: true, id: result.insertedId};
      } catch (error) {
        return {success: false, error: error.message, item: item};
      }
    });
    
    let batchResults = await Promise.all(batchPromises);
    
    // Collect results and track progress
    for (let result of batchResults) {
      if (result.success) {
        results.push(result);
      } else {
        errors.push(result);
      }
      processed++;
    }
    
    // Log progress
    console.log(`Processed ${processed}/${items.length} items`);
    
    // Optional: Update progress in database
    await SF.collection.updateOne("batch_jobs", 
      {_id: jobId}, 
      {
        processed: processed,
        total: items.length,
        progress: Math.round((processed / items.length) * 100)
      }
    );
  }
  
  return {
    total: items.length,
    successful: results.length,
    failed: errors.length,
    errors: errors
  };
}

3. Complex Workflow Orchestration

// Complex workflow with multiple async operations
async function processCompleteOrder(orderData) {
  let workflow = {
    orderId: null,
    customerId: orderData.customerId,
    status: "started",
    steps: [],
    errors: []
  };
  
  try {
    // Step 1: Validate customer
    workflow.steps.push({step: "validate_customer", status: "running"});
    let customer = await SF.user.getUserById(orderData.customerId);
    if (!customer) {
      throw new Error("Customer not found");
    }
    workflow.steps[0].status = "completed";
    
    // Step 2: Check inventory (parallel for all items)
    workflow.steps.push({step: "check_inventory", status: "running"});
    let inventoryChecks = await Promise.all(
      orderData.items.map(item => 
        SF.collection.findOne("inventory", {productId: item.productId})
      )
    );
    
    let insufficientStock = inventoryChecks.filter((stock, index) => 
      !stock || stock.quantity < orderData.items[index].quantity
    );
    
    if (insufficientStock.length > 0) {
      throw new Error("Insufficient inventory for some items");
    }
    workflow.steps[1].status = "completed";
    
    // Step 3: Create order record
    workflow.steps.push({step: "create_order", status: "running"});
    let order = await SF.collection.insertOne("orders", {
      customerId: customer._id,
      items: orderData.items,
      status: "processing",
      createdAt: new Date(),
      total: orderData.items.reduce((sum, item) => sum + (item.price * item.quantity), 0)
    });
    workflow.orderId = order.insertedId;
    workflow.steps[2].status = "completed";
    
    // Step 4: Update inventory (sequential to avoid conflicts)
    workflow.steps.push({step: "update_inventory", status: "running"});
    for (let item of orderData.items) {
      await SF.collection.updateOne("inventory",
        {productId: item.productId},
        {$inc: {quantity: -item.quantity, reserved: item.quantity}}
      );
    }
    workflow.steps[3].status = "completed";
    
    // Step 5: Generate invoice
    workflow.steps.push({step: "generate_invoice", status: "running"});
    let invoiceNumber = await SF.utils.nextValue("invoice", 6);
    let invoice = await SF.file.generatePDFFromTemplate(null, invoiceTemplateId, {
      variables: {
        customer: customer,
        order: order,
        invoiceNumber: invoiceNumber,
        items: orderData.items
      },
      SF_pdfName: `invoice-${invoiceNumber}.pdf`
    });
    workflow.steps[4].status = "completed";
    
    // Step 6: Send confirmation email
    workflow.steps.push({step: "send_confirmation", status: "running"});
    await SF.utils.axios.post("https://api.email-service.com/send", {
      to: customer.email,
      template: "order_confirmation",
      data: {
        customerName: customer.firstName,
        orderId: workflow.orderId,
        invoiceUrl: invoice.downloadUrl
      }
    });
    workflow.steps[5].status = "completed";
    
    workflow.status = "completed";
    
  } catch (error) {
    workflow.status = "failed";
    workflow.errors.push({
      message: error.message,
      step: workflow.steps.findIndex(s => s.status === "running"),
      timestamp: new Date()
    });
    
    // Rollback if necessary
    if (workflow.orderId) {
      await SF.collection.updateOne("orders", 
        {_id: workflow.orderId}, 
        {status: "failed", error: error.message}
      );
    }
  }
  
  // Log workflow completion
  await SF.collection.insertOne("workflow_logs", workflow);
  
  return workflow;
}

Common Pitfalls and How to Avoid Them

1. Forgetting to Use Await

// ❌ Wrong - Missing await
let user = SF.user.getUserById(userId); // This is a Promise, not the user data
console.log(user.firstName); // Error: Cannot read property 'firstName' of undefined

// ✅ Correct
let user = await SF.user.getUserById(userId);
console.log(user.firstName); // Works correctly

2. Using Async Operations in Array Methods Incorrectly

// ❌ Wrong - map() doesn't wait for async operations
let users = userIds.map(async (id) => {
  return await SF.user.getUserById(id);
}); // This returns an array of Promises, not users

// ✅ Correct - Use Promise.all()
let users = await Promise.all(
  userIds.map(id => SF.user.getUserById(id))
);

3. Not Handling Errors Properly

// ❌ Risky - Unhandled errors can crash the script
let result = await SF.collection.insertOne("orders", orderData);

// ✅ Safe - Always handle potential errors
try {
  let result = await SF.collection.insertOne("orders", orderData);
  return {success: true, orderId: result.insertedId};
} catch (error) {
  console.error('Order creation failed:', error);
  return {success: false, error: error.message};
}

4. Inefficient Sequential Processing

// ❌ Slow - Processing items one by one
for (let item of items) {
  await processItem(item); // Each item waits for the previous one
}

// ✅ Fast - Process items in parallel (when possible)
await Promise.all(items.map(item => processItem(item)));

// ✅ Balanced - Process in batches to avoid overwhelming the system
for (let i = 0; i < items.length; i += 10) {
  let batch = items.slice(i, i + 10);
  await Promise.all(batch.map(item => processItem(item)));
}

Testing and Debugging Tips

1. Use Console Logging Effectively

console.log('Starting order processing for:', orderId);

let order = await SF.collection.findOne("orders", {_id: orderId});
console.log('Retrieved order:', JSON.stringify(order, null, 2));

if (!order) {
  console.error('Order not found:', orderId);
  return {error: "Order not found"};
}

console.log('Order processing completed successfully');

2. Return Detailed Information for Debugging

return {
  success: true,
  orderId: order._id,
  customerEmail: customer.email,
  itemsProcessed: items.length,
  totalAmount: total,
  processingTime: Date.now() - startTime,
  // Include debug info in development
  debug: {
    inventoryChecks: inventoryResults,
    emailSent: emailResult.success,
    invoiceGenerated: !!invoiceFile
  }
};

3. Use the SoftyFlow Testing Interface

Add test variables with realistic sample data
Test each script component separately
Verify variable types and values
Check error handling paths

Performance Considerations

Minimize Database Queries: Use aggregation and bulk operations when possible
Use Parallel Processing: Run independent operations concurrently
Implement Timeouts: Set reasonable timeouts for external API calls
Cache Frequently Used Data: Store commonly accessed data in variables
Batch Operations: Process multiple items together rather than individually
Monitor Script Execution Time: Keep scripts lean and efficient

Conclusion

Mastering async/await patterns in SoftyFlow allows you to build robust, efficient, and maintainable process automation scripts. Remember to:

Always use await with SDK methods
Handle errors gracefully with try-catch blocks
Use parallel processing for independent operations
Implement proper logging and debugging practices
Test thoroughly using the SoftyFlow testing interface

Following these patterns and best practices will help you create reliable automation scripts that scale with your business processes.

Node.js Async Programming in SoftyFlow

Understanding Asynchronous Programming in SoftyFlow​

The Promise-Based Architecture​

Async/Await Fundamentals​

Basic Async/Await Pattern​

Sequential vs Parallel Execution​

Sequential Execution (Operations Depend on Each Other)​

Parallel Execution (Independent Operations)​

Common SoftyFlow Async Patterns​

1. Database Operations​

Single Document Operations​

Bulk Operations​

2. API Integration Patterns​

3. File Processing Patterns​

Best Practices for SoftyFlow Scripts​

1. Error Handling​

2. Performance Optimization​

Use Parallel Operations When Possible​

Minimize Database Queries​

3. Variable Management​

4. Conditional Processing​

Advanced Patterns​

1. Retry Logic for External Services​

2. Batch Processing with Progress Tracking​

3. Complex Workflow Orchestration​

Common Pitfalls and How to Avoid Them​

1. Forgetting to Use Await​

2. Using Async Operations in Array Methods Incorrectly​

3. Not Handling Errors Properly​

4. Inefficient Sequential Processing​

Testing and Debugging Tips​

1. Use Console Logging Effectively​

2. Return Detailed Information for Debugging​

3. Use the SoftyFlow Testing Interface​

Performance Considerations​

Conclusion​