Node.js Async Programming in Softyflow

This guide explains how to effectively use Node.js asynchronous programming patterns in Softyflow scripts, with focus on async/await syntax and best practices for writing robust, performant scripts.

Understanding Asynchronous Programming in SoftyflowDirect link to Understanding Asynchronous Programming in Softyflow

Softyflow's JavaScript execution environment is built on Node.js, which means all I/O operations (database queries, API calls, file operations) are asynchronous. Understanding how to properly handle asynchronous operations is crucial for writing effective scripts.

The Promise-Based ArchitectureDirect link to The Promise-Based Architecture

All Softyflow SDK methods return Promises, which represent eventual completion (or failure) of asynchronous operations:

// ❌ Wrong - This won't work as expected
let users = SF.collection.find("users", {status: "active"}); // This returns a Promise object, not the data

// ✅ Correct - Using await to get the actual data
let users = await SF.collection.find("users", {status: "active"});

Async/Await FundamentalsDirect link to Async/Await Fundamentals

Basic Async/Await PatternDirect link to Basic Async/Await Pattern

The await keyword can only be used in functions marked with async. In Softyflow scripts, the execution context is automatically async, so you can use await directly:

// Softyflow script - automatically in async context
let user = await SF.user.getUserById("6042551d231e8c8ade19bc09");
let orders = await SF.collection.find("orders", {userId: user._id});

return {
  user: user,
  orderCount: orders.length,
  totalAmount: orders.reduce((sum, order) => sum + order.amount, 0)
};

Sequential vs Parallel ExecutionDirect link to Sequential vs Parallel Execution

Understanding when operations run sequentially vs in parallel is crucial for performance:

Sequential Execution (Operations Depend on Each Other)Direct link to Sequential Execution (Operations Depend on Each Other)

// ✅ Good - When each operation depends on the previous one
let user = await SF.user.getUserById(userId);
let userGroup = await SF.user.getGroupById(user.groupId);
let permissions = await SF.collection.findOne("permissions", {groupId: userGroup._id});

return {
  user: user,
  group: userGroup,
  permissions: permissions
};

Parallel Execution (Independent Operations)Direct link to Parallel Execution (Independent Operations)

// ✅ Better - When operations are independent, run them in parallel
let [user, recentOrders, supportTickets] = await Promise.all([
  SF.user.getUserById(userId),
  SF.collection.find("orders", {userId: userId, createdAt: {$gte: lastMonth}}),
  SF.collection.find("tickets", {userId: userId, status: "open"})
]);

return {
  user: user,
  recentOrdersCount: recentOrders.length,
  openTicketsCount: supportTickets.length
};

Common Softyflow Async PatternsDirect link to Common Softyflow Async Patterns

1. Database OperationsDirect link to 1. Database Operations

Single Document OperationsDirect link to Single Document Operations

// Find and update pattern
let existingRecord = await SF.collection.findOne("customers", {email: customerEmail});

if (existingRecord) {
  // Update existing customer
  await SF.collection.updateOne("customers", 
    {email: customerEmail}, 
    {lastLogin: new Date(), loginCount: existingRecord.loginCount + 1}
  );
} else {
  // Create new customer
  await SF.collection.insertOne("customers", {
    email: customerEmail,
    createdAt: new Date(),
    loginCount: 1
  });
}

Bulk OperationsDirect link to Bulk Operations

// Efficient bulk processing
let usersToUpdate = await SF.collection.find("users", {status: "pending"});

// Prepare bulk operations
let bulkOps = usersToUpdate.map(user => ({
  updateOne: {
    filter: {_id: user._id},
    update: {$set: {status: "active", activatedAt: new Date()}}
  }
}));

// Execute all updates in one operation
if (bulkOps.length > 0) {
  await SF.collection.bulkWrite("users", bulkOps);
}

return {processed: usersToUpdate.length};

2. API Integration PatternsDirect link to 2. API Integration Patterns

// External API call with error handling
try {
  let response = await SF.utils.axios.get(`https://api.external-service.com/users/${userId}`, {
    headers: {
      'Authorization': `Bearer ${apiToken}`,
      'Content-Type': 'application/json'
    },
    timeout: 5000 // 5 second timeout
  });
  
  // Process the response
  let externalUser = response.data;
  
  // Update local database with external data
  await SF.collection.updateOne("users", 
    {_id: userId}, 
    {
      externalId: externalUser.id,
      lastSyncAt: new Date(),
      syncStatus: "success"
    }
  );
  
  return {success: true, externalId: externalUser.id};
  
} catch (error) {
  // Handle API errors gracefully
  console.error('External API call failed:', error.message);
  
  // Update sync status
  await SF.collection.updateOne("users", 
    {_id: userId}, 
    {syncStatus: "failed", lastSyncError: error.message}
  );
  
  return {success: false, error: error.message};
}

3. File Processing PatternsDirect link to 3. File Processing Patterns

// Generate PDF and save to collection
try {
  // Get template and data in parallel
  let [template, customerData] = await Promise.all([
    SF.file.getFileContentById(templateId),
    SF.collection.findOne("customers", {_id: customerId})
  ]);
  
  // Generate PDF
  let pdfFile = await SF.file.generatePDFFromTemplate(null, templateId, {
    variables: {
      customer: customerData,
      generatedAt: new Date().toLocaleDateString(),
      invoiceNumber: await SF.utils.nextValue("invoice", 6)
    },
    SF_pdfName: `invoice-${customerData.code}.pdf`
  });
  
  // Save reference in orders collection
  await SF.collection.updateOne("orders", 
    {customerId: customerId}, 
    {pdfFileId: pdfFile._id, pdfGeneratedAt: new Date()}
  );
  
  return {pdfGenerated: true, fileId: pdfFile._id};
  
} catch (error) {
  console.error('PDF generation failed:', error);
  return {pdfGenerated: false, error: error.message};
}

Best Practices for Softyflow ScriptsDirect link to Best Practices for Softyflow Scripts

1. Error HandlingDirect link to 1. Error Handling

Always wrap potentially failing operations in try-catch blocks:

// ✅ Good - Robust error handling
async function processOrder(orderId) {
  try {
    let order = await SF.collection.findOne("orders", {_id: orderId});
    
    if (!order) {
      throw new Error(`Order ${orderId} not found`);
    }
    
    // Process order...
    let result = await processPayment(order);
    
    return {success: true, result: result};
    
  } catch (error) {
    console.error('Order processing failed:', error.message);
    
    // Log error for debugging
    await SF.collection.insertOne("error_logs", {
      operation: "processOrder",
      orderId: orderId,
      error: error.message,
      timestamp: new Date()
    });
    
    return {success: false, error: error.message};
  }
}

2. Performance OptimizationDirect link to 2. Performance Optimization

Use Parallel Operations When PossibleDirect link to Use Parallel Operations When Possible

// ❌ Slow - Sequential operations
let user = await SF.user.getUserById(userId);
let orders = await SF.collection.find("orders", {userId: userId});
let preferences = await SF.collection.findOne("preferences", {userId: userId});

// ✅ Fast - Parallel operations
let [user, orders, preferences] = await Promise.all([
  SF.user.getUserById(userId),
  SF.collection.find("orders", {userId: userId}),
  SF.collection.findOne("preferences", {userId: userId})
]);

Minimize Database QueriesDirect link to Minimize Database Queries

// ❌ Inefficient - Multiple queries
let results = [];
for (let userId of userIds) {
  let user = await SF.user.getUserById(userId);
  results.push(user);
}

// ✅ Efficient - Single aggregation query
let results = await SF.collection.aggregate("users", [
  {$match: {_id: {$in: userIds}}},
  {$project: {firstName: 1, lastName: 1, email: 1}}
]);

3. Variable ManagementDirect link to 3. Variable Management

// ✅ Good - Clear variable organization
let processData = {
  // Input data validation
  customerId: customerId || null,
  orderDate: orderDate || new Date(),
  items: Array.isArray(items) ? items : [],
  
  // Calculated values
  subtotal: 0,
  tax: 0,
  total: 0,
  
  // Status tracking
  processed: false,
  errors: []
};

// Validate required data
if (!processData.customerId) {
  processData.errors.push("Customer ID is required");
}

if (processData.items.length === 0) {
  processData.errors.push("Order must contain at least one item");
}

if (processData.errors.length > 0) {
  return processData; // Return early with errors
}

// Process the order...

4. Conditional ProcessingDirect link to 4. Conditional Processing

// ✅ Good - Clean conditional logic with async operations
let approvalRequired = false;
let approvers = [];

// Check if approval is needed based on amount
if (orderTotal > 10000) {
  approvalRequired = true;
  approvers = await SF.user.getUsersInGroupList(["managers", "directors"]);
} else if (orderTotal > 5000) {
  approvalRequired = true;
  approvers = await SF.user.getUsersinGroup("managers");
}

if (approvalRequired) {
  // Create approval task
  let approvalTask = await SF.instance.validateTask(currentTaskId, {
    requiresApproval: true,
    approvers: approvers.map(user => user.email),
    amount: orderTotal
  });
  
  return {
    status: "pending_approval",
    approvers: approvers.length,
    taskId: approvalTask.id
  };
} else {
  // Auto-approve small orders
  return {
    status: "approved",
    autoApproved: true
  };
}

Advanced PatternsDirect link to Advanced Patterns

1. Retry Logic for External ServicesDirect link to 1. Retry Logic for External Services

async function callExternalServiceWithRetry(url, data, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      let response = await SF.utils.axios.post(url, data, {
        timeout: 10000
      });
      
      return response.data; // Success, return data
      
    } catch (error) {
      console.log(`Attempt ${attempt} failed:`, error.message);
      
      if (attempt === maxRetries) {
        throw new Error(`Failed after ${maxRetries} attempts: ${error.message}`);
      }
      
      // Wait before retry (exponential backoff)
      await new Promise(resolve => setTimeout(resolve, 1000 * attempt));
    }
  }
}

// Usage
try {
  let result = await callExternalServiceWithRetry("https://api.example.com/process", orderData);
  return {success: true, externalId: result.id};
} catch (error) {
  return {success: false, error: error.message};
}

2. Batch Processing with Progress TrackingDirect link to 2. Batch Processing with Progress Tracking

async function processBatchWithProgress(items, batchSize = 10) {
  let results = [];
  let processed = 0;
  let errors = [];
  
  // Process items in batches
  for (let i = 0; i < items.length; i += batchSize) {
    let batch = items.slice(i, i + batchSize);
    
    // Process batch in parallel
    let batchPromises = batch.map(async (item) => {
      try {
        let result = await SF.collection.insertOne("processed_items", {
          originalId: item.id,
          processedAt: new Date(),
          data: item
        });
        return {success: true, id: result.insertedId};
      } catch (error) {
        return {success: false, error: error.message, item: item};
      }
    });
    
    let batchResults = await Promise.all(batchPromises);
    
    // Collect results and track progress
    for (let result of batchResults) {
      if (result.success) {
        results.push(result);
      } else {
        errors.push(result);
      }
      processed++;
    }
    
    // Log progress
    console.log(`Processed ${processed}/${items.length} items`);
    
    // Optional: Update progress in database
    await SF.collection.updateOne("batch_jobs", 
      {_id: jobId}, 
      {
        processed: processed,
        total: items.length,
        progress: Math.round((processed / items.length) * 100)
      }
    );
  }
  
  return {
    total: items.length,
    successful: results.length,
    failed: errors.length,
    errors: errors
  };
}

3. Complex Workflow OrchestrationDirect link to 3. Complex Workflow Orchestration

// Complex workflow with multiple async operations
async function processCompleteOrder(orderData) {
  let workflow = {
    orderId: null,
    customerId: orderData.customerId,
    status: "started",
    steps: [],
    errors: []
  };
  
  try {
    // Step 1: Validate customer
    workflow.steps.push({step: "validate_customer", status: "running"});
    let customer = await SF.user.getUserById(orderData.customerId);
    if (!customer) {
      throw new Error("Customer not found");
    }
    workflow.steps[0].status = "completed";
    
    // Step 2: Check inventory (parallel for all items)
    workflow.steps.push({step: "check_inventory", status: "running"});
    let inventoryChecks = await Promise.all(
      orderData.items.map(item => 
        SF.collection.findOne("inventory", {productId: item.productId})
      )
    );
    
    let insufficientStock = inventoryChecks.filter((stock, index) => 
      !stock || stock.quantity < orderData.items[index].quantity
    );
    
    if (insufficientStock.length > 0) {
      throw new Error("Insufficient inventory for some items");
    }
    workflow.steps[1].status = "completed";
    
    // Step 3: Create order record
    workflow.steps.push({step: "create_order", status: "running"});
    let order = await SF.collection.insertOne("orders", {
      customerId: customer._id,
      items: orderData.items,
      status: "processing",
      createdAt: new Date(),
      total: orderData.items.reduce((sum, item) => sum + (item.price * item.quantity), 0)
    });
    workflow.orderId = order.insertedId;
    workflow.steps[2].status = "completed";
    
    // Step 4: Update inventory (sequential to avoid conflicts)
    workflow.steps.push({step: "update_inventory", status: "running"});
    for (let item of orderData.items) {
      await SF.collection.updateOne("inventory",
        {productId: item.productId},
        {$inc: {quantity: -item.quantity, reserved: item.quantity}}
      );
    }
    workflow.steps[3].status = "completed";
    
    // Step 5: Generate invoice
    workflow.steps.push({step: "generate_invoice", status: "running"});
    let invoiceNumber = await SF.utils.nextValue("invoice", 6);
    let invoice = await SF.file.generatePDFFromTemplate(null, invoiceTemplateId, {
      variables: {
        customer: customer,
        order: order,
        invoiceNumber: invoiceNumber,
        items: orderData.items
      },
      SF_pdfName: `invoice-${invoiceNumber}.pdf`
    });
    workflow.steps[4].status = "completed";
    
    // Step 6: Send confirmation email
    workflow.steps.push({step: "send_confirmation", status: "running"});
    await SF.utils.axios.post("https://api.email-service.com/send", {
      to: customer.email,
      template: "order_confirmation",
      data: {
        customerName: customer.firstName,
        orderId: workflow.orderId,
        invoiceUrl: invoice.downloadUrl
      }
    });
    workflow.steps[5].status = "completed";
    
    workflow.status = "completed";
    
  } catch (error) {
    workflow.status = "failed";
    workflow.errors.push({
      message: error.message,
      step: workflow.steps.findIndex(s => s.status === "running"),
      timestamp: new Date()
    });
    
    // Rollback if necessary
    if (workflow.orderId) {
      await SF.collection.updateOne("orders", 
        {_id: workflow.orderId}, 
        {status: "failed", error: error.message}
      );
    }
  }
  
  // Log workflow completion
  await SF.collection.insertOne("workflow_logs", workflow);
  
  return workflow;
}

Common Pitfalls and How to Avoid ThemDirect link to Common Pitfalls and How to Avoid Them

1. Forgetting to Use AwaitDirect link to 1. Forgetting to Use Await

// ❌ Wrong - Missing await
let user = SF.user.getUserById(userId); // This is a Promise, not the user data
console.log(user.firstName); // Error: Cannot read property 'firstName' of undefined

// ✅ Correct
let user = await SF.user.getUserById(userId);
console.log(user.firstName); // Works correctly

2. Using Async Operations in Array Methods IncorrectlyDirect link to 2. Using Async Operations in Array Methods Incorrectly

// ❌ Wrong - map() doesn't wait for async operations
let users = userIds.map(async (id) => {
  return await SF.user.getUserById(id);
}); // This returns an array of Promises, not users

// ✅ Correct - Use Promise.all()
let users = await Promise.all(
  userIds.map(id => SF.user.getUserById(id))
);

3. Not Handling Errors ProperlyDirect link to 3. Not Handling Errors Properly

// ❌ Risky - Unhandled errors can crash the script
let result = await SF.collection.insertOne("orders", orderData);

// ✅ Safe - Always handle potential errors
try {
  let result = await SF.collection.insertOne("orders", orderData);
  return {success: true, orderId: result.insertedId};
} catch (error) {
  console.error('Order creation failed:', error);
  return {success: false, error: error.message};
}

4. Inefficient Sequential ProcessingDirect link to 4. Inefficient Sequential Processing

// ❌ Slow - Processing items one by one
for (let item of items) {
  await processItem(item); // Each item waits for the previous one
}

// ✅ Fast - Process items in parallel (when possible)
await Promise.all(items.map(item => processItem(item)));

// ✅ Balanced - Process in batches to avoid overwhelming the system
for (let i = 0; i < items.length; i += 10) {
  let batch = items.slice(i, i + 10);
  await Promise.all(batch.map(item => processItem(item)));
}

Testing and Debugging TipsDirect link to Testing and Debugging Tips

1. Use Console Logging EffectivelyDirect link to 1. Use Console Logging Effectively

console.log('Starting order processing for:', orderId);

let order = await SF.collection.findOne("orders", {_id: orderId});
console.log('Retrieved order:', JSON.stringify(order, null, 2));

if (!order) {
  console.error('Order not found:', orderId);
  return {error: "Order not found"};
}

console.log('Order processing completed successfully');

2. Return Detailed Information for DebuggingDirect link to 2. Return Detailed Information for Debugging

return {
  success: true,
  orderId: order._id,
  customerEmail: customer.email,
  itemsProcessed: items.length,
  totalAmount: total,
  processingTime: Date.now() - startTime,
  // Include debug info in development
  debug: {
    inventoryChecks: inventoryResults,
    emailSent: emailResult.success,
    invoiceGenerated: !!invoiceFile
  }
};

3. Use the Softyflow Testing InterfaceDirect link to 3. Use the Softyflow Testing Interface

Add test variables with realistic sample data
Test each script component separately
Verify variable types and values
Check error handling paths

Performance ConsiderationsDirect link to Performance Considerations

Minimize Database Queries: Use aggregation and bulk operations when possible
Use Parallel Processing: Run independent operations concurrently
Implement Timeouts: Set reasonable timeouts for external API calls
Cache Frequently Used Data: Store commonly accessed data in variables
Batch Operations: Process multiple items together rather than individually
Monitor Script Execution Time: Keep scripts lean and efficient

ConclusionDirect link to Conclusion

Mastering async/await patterns in Softyflow allows you to build robust, efficient, and maintainable process automation scripts. Remember to:

Always use await with SDK methods
Handle errors gracefully with try-catch blocks
Use parallel processing for independent operations
Implement proper logging and debugging practices
Test thoroughly using the Softyflow testing interface

Following these patterns and best practices will help you create reliable automation scripts that scale with your business processes.

Node.js Async Programming in Softyflow

Understanding Asynchronous Programming in Softyflow#Direct link to Understanding Asynchronous Programming in Softyflow

The Promise-Based Architecture#Direct link to The Promise-Based Architecture

Async/Await Fundamentals#Direct link to Async/Await Fundamentals

Basic Async/Await Pattern#Direct link to Basic Async/Await Pattern

Sequential vs Parallel Execution#Direct link to Sequential vs Parallel Execution

Sequential Execution (Operations Depend on Each Other)#Direct link to Sequential Execution (Operations Depend on Each Other)

Parallel Execution (Independent Operations)#Direct link to Parallel Execution (Independent Operations)

Common Softyflow Async Patterns#Direct link to Common Softyflow Async Patterns

1. Database Operations#Direct link to 1. Database Operations

Single Document Operations#Direct link to Single Document Operations

Bulk Operations#Direct link to Bulk Operations

2. API Integration Patterns#Direct link to 2. API Integration Patterns

3. File Processing Patterns#Direct link to 3. File Processing Patterns

Best Practices for Softyflow Scripts#Direct link to Best Practices for Softyflow Scripts

1. Error Handling#Direct link to 1. Error Handling

2. Performance Optimization#Direct link to 2. Performance Optimization

Use Parallel Operations When Possible#Direct link to Use Parallel Operations When Possible

Minimize Database Queries#Direct link to Minimize Database Queries

3. Variable Management#Direct link to 3. Variable Management

4. Conditional Processing#Direct link to 4. Conditional Processing

Advanced Patterns#Direct link to Advanced Patterns

1. Retry Logic for External Services#Direct link to 1. Retry Logic for External Services

2. Batch Processing with Progress Tracking#Direct link to 2. Batch Processing with Progress Tracking

3. Complex Workflow Orchestration#Direct link to 3. Complex Workflow Orchestration

Common Pitfalls and How to Avoid Them#Direct link to Common Pitfalls and How to Avoid Them

1. Forgetting to Use Await#Direct link to 1. Forgetting to Use Await

2. Using Async Operations in Array Methods Incorrectly#Direct link to 2. Using Async Operations in Array Methods Incorrectly

3. Not Handling Errors Properly#Direct link to 3. Not Handling Errors Properly

4. Inefficient Sequential Processing#Direct link to 4. Inefficient Sequential Processing

Testing and Debugging Tips#Direct link to Testing and Debugging Tips

1. Use Console Logging Effectively#Direct link to 1. Use Console Logging Effectively

2. Return Detailed Information for Debugging#Direct link to 2. Return Detailed Information for Debugging

3. Use the Softyflow Testing Interface#Direct link to 3. Use the Softyflow Testing Interface

Performance Considerations#Direct link to Performance Considerations

Conclusion#Direct link to Conclusion