Wednesday, June 1, 2016

Scenarios to Understand Subscription Tier Throttling in WSO2 API Manager 2.0


  • With the new throttling implementation, the subscription level tier quota will define the limit that particular application can access that API 

  • So basically the throttling tier policy is configured for subscription level tier by : appId + apiName + version. 
  • This can be defined as per request count or bandwidth. 
  • If application is used by 1000 users and subscribe to a 50000 Req/Min tier, then all the subscribed 1000 users can invoke maximum of 50000 Request per minute since it does not consider the user identification in subscription level tier policy.
  • With the previous throttling implementation any application user could access limit of 50000 Req/Min. 

  • When configuring Subscription level tier, Burst/rate limiting is also introduced to control the maximum requests that can be sent by a particular subscriber for a given period.
  • So if the burst limit is configured as 1000 Request/s, each user will be able to send only 1000 requests per second as maximum until it reaches 50000 requests for that minute.
  • If there are 10 application users using subscribed using 10 different applications, each user get 50000 requests with the limitation of sending burst requests 1000 per second.


Scenario 1 – Different users using same application using their user tokens


Throttle out an API by a subscription tier when few users from same tenant invoke a particular API subscribed via the same application when quota limit is 'request count' and when there is no burst limit


Preconditions:

1. API Manager should be up and running and user1 and user2 should be signed up with subscriber permission.
2. A subscription tier should be created as below.
  • Tier Name : Silver
  • Request Count : 2000
  • Unit Time : 1 minute
  • Burst Control (Rate Limiting) : 0
  • Stop on Quota Reach : Not selected
  • Billing Plan : Free or Commercial
  • Custom Attributes : None
  • Permissions : not defined
3. API 1 should have been been created and published as below by a publisher,
  • Subscription Tiers : Silver
  • GET resource level Tier : Unlimited is set
4. A developer subscribe to the API1
  • Application created with an Unlimited Tier. app1
  • Subscribe using application with Silver
5. Generate production keys for the particular application app1 and retrieve consumer key and secret.
6. User1 and User2 should generate their user tokens using the consumer key and secret generated in the above step.
User1 using app1 :

User1 Token = curl -k -d "grant_type=password&username=<Username>
&password=<Password>" -H "Authorization: Basic <app1_Token>" 
https://192.168.124.1:8243/token
 
 
User2 using app1 :

User2 Token = curl -k -d "grant_type=password&username=<Username>
&password=<Password>" -H "Authorization: Basic <app1_Token>" 
https://192.168.124.1:8243/token
 
 
Authorization: Basic <app1_token> =
<Base64encode(consumerkey:consumer secret of app1)>

Step Expected Result
User 1 and User 2 invoke the GET resource as below within a minute using their user token
  • User1 : 900 using user1 token
  • User2 : 1101 using user2 token
User who exceeds the 2000th request should be notified as throttled out.


Scenario 2 : Same user using different applications using their user tokens



Throttle out an API by a subscription tier when the same user invokes a particular API subscribed via different applications when quota limit is 'request count' and when there is no burst limit.



Preconditions:

1. API Manager should be up and running and user1 should be signed up with subscriber permission.
2. A subscription tier should be created as below.
  • Tier Name : Silver
  • Request Count : 2000
  • Unit Time : 1 minute
  • Burst Control (Rate Limiting) : 0
  • Stop on Quota Reach : Not selected
  • Billing Plan : Free or Commercial
  • Custom Attributes : None
  • Permissions : not defined
3. API 1 should have been been created and published as below by a publisher,
  • Subscription Tiers : Silver
  • GET resource level Tier : Unlimited is set
4. A developer subscribe to the API1 
  • 2 Applications created with an Unlimited Tier. app1 and app2
  • Subscribe API1 using applications (app1 and app2) with Silver
5. Generate production keys for the paticular applications app1 and app2 and retrieve consumer key and secret.
6. User1 should generate the user tokens using the consumer key and secret generated in the above step for both apps

User1 using app1 :

User1 Token1 = curl -k -d "grant_type=password&username=<Username>&password=<Password>"
 -H "Authorization: Basic <app1_Token>" https://192.168.124.1:8243/token
 
 
User1 using app2 :

User1 Token2 = curl -k -d "grant_type=password&username=<Username>&password=<Password>"
 -H "Authorization: Basic <app2_Token>" https://192.168.124.1:8243/token
 
 
Authorization: Basic <app1_token>
= <Base64encode(consumerkey:consumer secret of app1)>

Authorization: Basic <app2_token>
 = <Base64encode(consumerkey:consumer secret of app2)>


Step Expected Result
User 1 invoke the GET resource as below within a minute using
user token1 and token2

  • 900 requests using user1 token1
  • 1101 requests using user1 token2
User will be able to invoke successfully all the requests
User 1 invokes the GET resource as below within a minute using their user token1 and token2

  • 2000 using user1 token1 
  • 2001 using user1 token2
When user1 invokes the 2001st request using token2, will be notified as throttled out while other requests will be successful.



Scenario 3 : Different users via different applications using their user tokens


Throttle out an API by a subscription tier when few users from same tenant invoke a particular API subscribed via different applications when quota limit is request count and when there is no burst limit.

Preconditions:

1. API Manager should be up and running and user1 and user2 should be signed up with subscriber permission.
2. A subscription tier should be created as below.
  • Tier Name : Silver
  • Request Count : 2000
  • Unit Time : 1 minute
  • Burst Control (Rate Limiting) : 0
  • Stop on Quota Reach : Not selected
  • Billing Plan : Free or Commercial
  • Custom Attributes : None
  • Permissions : not defined
3. API 1 should have been been created and published as below by a publisher,
  • Subscription Tiers : Silver
  • GET resource level Tier : Unlimited is set
4. A developer subscribe to the API1
  • Application 1 and application 2 created with an Unlimited Tier. app1 and app2
  • Subscribe using applications with Silver
5. Generate production keys for the paticular applications app1 and app2 for 2 different users and retrieve consumer key and secret.
6. User1 and User2 should generate their user tokens using the consumer key and secret generated in the above step.

User 1 using app1 :

User token 1 = curl -k -d "grant_type=password&username=user1&password=user1" 
-H "Authorization: Basic <app1_Token>" https://192.168.124.1:8243/token
 
 
User 2 using app2 :

User token 2 = curl -k -d "grant_type=password&username=user2&password=user2" 
-H "Authorization: Basic <app2_Token>" https://192.168.124.1:8243/token


Authorization: Basic <app1_Token>
 = <Base64encode(consumerkey:consumer secret of app1)>

Authorization: Basic <app2_Token>
= <Base64encode(consumerkey:consumer secret of app2)>

Step Expected Result
User 1 and User 2 invoke the GET resource as below within a minute using their user token

  • User1 : 900 using user1 token 
  • User2 : 1101 using user2 token
Both users will be able to invoke successfully.
User 1 and User 2 invoke the GET resource as below within a minute using their user tokens

  • User1 : 2000 using user1 token 
  • User2 : 2000 using user2 token
Both users will be able to invoke successfully.
User 1 and User 2 invoke the GET resource as below within a minute using their user tokens

  • User1 : 2001 requests using user1 token 
  • User2 : 2001 requests using user2 token
Both users will be notified as throttled out.



Scenario 4 : Different users via same application using test access token


Throttle out an API by a subscription tier when few users from same tenant invoke a paticular API subscribed via the same application and test access token (grant_type = client credentials) when quota limit is 'request count' and when there is no burst limit



Preconditions :
1. API Manager should be up and running and user1 and user2 should be signed up with subscriber permission.
2. A subscription tier should be created as below.
  • Tier Name : Silver
  • Request Count : 2000
  • Unit Time : 1 minute
  • Burst Control (Rate Limiting) : 0
  • Stop on Quota Reach : Not selected
  • Billing Plan : Free or Commercial
  • Custom Attributes : None
  • Permissions : not defined
3. API 1 should have been been created and published as below by a publisher,
  • Subscription Tiers : Silver
  • GET resource level Tier : Unlimited is set
4. A developer subscribe to the API1 
  • Application created with an Unlimited Tier. app1
  • Subscribe using application with Silver
5. Generate production keys for the paticular application app1 and retrieve test access token.
6. Test access token can be retrieved via below command.

Developer generates an access token using app1 :

Test Access Token = curl -k -d "grant_type=client_credentials
-H "Authorization: Basic <app1_Token>" https://192.168.124.1:8243/token

Authorization: Basic <app1_token>
 = <Base64encode(consumerkey:consumer secret of app1)>

Step Expected Result
User 1 and User 2 invoke the GET resource as below within a minute using the same test access token generated

  • User1 : 900 requests using test access token 
  • User2 : 1100 requests using test access token
Both users will be able to invoke successfully.
User 1 and User 2 invoke the GET resource as below within a minute using the same test access token

  • User1 : 900 using test access token 
  • User2 : 1101 using test access token
User who exceeds the 2000th request should be notified as throttled out.

Scenario 5: Different users via same application via user token when the burst limit is configured


Throttle out an API by a subscription tier when few users from same tenant invoke a particular API subscribed via same applications when quota limit is request count and when there is a burst limit configured.

Preconditions :

1. API Manager should be up and running and user1 and user2 should be signed up with subscriber permission.
2. A subscription tier should be created as below.
  • Tier Name : Silver
  • Request Count : 2000
  • Unit Time : 1 hour
  • Burst Control (Rate Limiting) : 100 Request/m
  • Stop on Quota Reach : Not selected
  • Billing Plan : Free or Commercial
  • Custom Attributes : None
  • Permissions : not defined
3. API 1 should have been been created and published as below by a publisher,
  • Subscription Tiers : Silver
  • GET resource level Tier : Unlimited is set
4. A developer subscribe to the API1 
  • Application 1 created with an Unlimited Tier. app1
  • Subscribe using applications with Silver
5. Generate production keys for the particular applications app1 and app2 for 2 different users and retrieve consumer key and secret.
6. User1 and User2 should generate their user tokens using the consumer key and secret generated in the above step.

User 1 using app1 :

User token 1 = curl -k -d "grant_type=password&username=user1&password=user1" 
-H "Authorization: Basic <app1_Token>" https://192.168.124.1:8243/token
 
 
User 2 using app1 :

User token 2 = curl -k -d "grant_type=password&username=user2&password=user2" 
-H "Authorization: Basic <app1_Token>" https://192.168.124.1:8243/token

Authorization: Basic <app1_Token>
 = <Base64encode(consumerkey:consumer secret of app1)>

Step Expected Result
User1 invoke with 100 requests (a burst) within a minute using their user token User should be able to invoke successfully
User 1 try to send a request within the same minute in step 2. User1 will be notified as you have exceeded your quota until the next minute
User 2 invoke with 100 requests (a burst) within the same minute in step 2 using user2's user token. User 2 will be able to invoke successfully.
User 2 invokes again within the same minute in step 2 using user2's user token User should be notified as exceeded quota until the next minute.
User 1 and User 2 invoke the GET resource as below within an hour with below requests sticking to burst limit. (100 requests/m)

  • User1 : 1000 requests using user1 token 
  • User2 : 1001 requests using user2 token
User who exceeds the throttling limit by sending the 2001st request will be notified as throttled out until the next hour since it is configured as 2000 req/hr.

Until that all the requests will be successfully invoked sticking to the burst limit.

Monday, May 30, 2016

Difference in API and User Level Advance Resource throttling in WSO2 API Manager 2.0

From WSO2 API Manager 2.0 throttling implementation onward 2 different throttling levels have been introduced in Resource Level throttling.

When you login to admin dashboard you can see Advance resource throttling tier configurations under throttle policies section as given in the below screenshot.




When you add a resource tier, you can select either API or resource level as below.


API Level Resource Tier


For API level policy, it is the shared quota of all applications that invoke the API.
if someone selects API Level policy then selecting resource level policy will be disabled.

So as an example if there are 2 users subscribe to the same api, the request count or bandwidth  defined to the API is applicable for both users as a shared quota. So if you have defined 10000 requests per minute both users can share that amount.

User Level Resource Tier

For User level policy, it is the quota assigned to each application that will invoke the API.
So the quota is assigned for single user who can access the particular API from multiple applications. Simply when it's user level, throttle key will be associate with user name.

So as an example if you have selected user level, and when there are 2 users subscribed to the same API, the defined count in tier will be assigned to each user. So if you have defined 10000 requests per minute, both users get 10000 requests /1 minute per each as a total of 20000 requests.

Monday, April 11, 2016

How to hide credentials used in mediation configuration using Secure Vault in WSO2 ESB

Eventhough we use secure vault to encrypt password, it is not possible to use secure vault directly in the mediation configuration. As an example imagine you need to hide a password given in a proxy.

All you have to do is using Secure Vault Password Management screen in WSO2 ESB.


1. Run sh ciphertool.sh -Dconfigure and enable secure vault
2. Start the WSO2 ESB with
3. Go to  Manage -> Secure Vault Tool and then click Manage Passwords
4. You will see the below screen.




5. Click on Add New Password to encrypt and store and specify values.

Vault Key - The alias for the password. E.g., sfpassword
Password -  The actual password.
Re-enter password - Re enter the password.

6. Once you save you will be able to save the newly created vault key.

7. Now you can use it in your proxy or any other mediation configuration as below.

E.g.,

   <inSequence>  
      <property name="password" expression="wso2:vault-lookup('sfpassword')"/>  
      <log level="custom">  
       <property name="InMsg" value="Invoking the proxy"/>  
       <property name="pickPassword" expression="get-property('password')"/>  
      </log>  
      <property name="OUT_ONLY" value="true"/>  
      <send/>  
    </inSequence>  


8. If you want to hide the salesforce password,

Salesforce Configuration :

  <salesforce.getUserInfo configKey="MySFConfig"/>   
    <salesforce.query>   
    <batchSize>1</batchSize>   
    <queryString>{$ctx:CompleteLeadQuery}</queryString>   
  </salesforce.query>   


MySFConfig with salesforce connection details :


 <salesforce.init>  
   <username>xxxx@abc.com</username>  
   <password>{wso2:vault-lookup('sfpassword')}</password>  
   <loginUrl>https://login.salesforce.com/services/Soap/u/27.0</loginUrl>  
   <blocking>false</blocking>  
 </salesforce.init>  



Tuesday, February 2, 2016

Simple WSO2 ESB API which Queries salesforce and build a json array using payloadFactory Mediator

1. Download WSO2 ESB and Salesforce connector.
2. Add the salesforce connector and enable it.

For more information please follow : https://docs.wso2.com/display/ESBCONNECTORS/Working+with+Salesforce+Connector+Operations


Then the below API can be used to query User object using profile id and then build a json object using payload factory mediator.

  • MySFConfig should have the required login information.
  • 00e90000001aVwiAAE is the profile id of a user.


 <api xmlns="http://ws.apache.org/ns/synapse" name="leads1" context="/leads1">  
   <resource methods="GET">  
    <inSequence>  
      <property name="LeadQuery" value="Select u.Username, u.ProfileId, u.Name, u.LastName, u.Email From User u where ProfileId='" scope="default" type="STRING"/>  
      <property name="Apostrophe" value="'" scope="default" type="STRING"/>  
      <property name="ProfileId" value="00e90000001aVwiAAE" scope="default" type="STRING"/>  
      <property name="CompleteLeadQuery" expression="fn:concat($ctx:LeadQuery, $ctx:ProfileId, $ctx:Apostrophe)" scope="default" type="STRING"/>  
      <salesforce.getUserInfo configKey="MySFConfig"/>  
      <salesforce.query>  
       <batchSize>1</batchSize>  
       <queryString>{$ctx:CompleteLeadQuery}</queryString>  
      </salesforce.query>  
      <property xmlns:ns="urn:partner.soap.sforce.com" xmlns:sf="urn:sobject.partner.soap.sforce.com" name="Name" expression="//ns:queryResponse/ns:result/ns:records/sf:Name/text()" scope="default" type="STRING"/>  
      <property xmlns:ns="urn:partner.soap.sforce.com" xmlns:sf="urn:sobject.partner.soap.sforce.com" name="Username" expression="//ns:queryResponse/ns:result/ns:records/sf:Username/text()" scope="default" type="STRING"/>  
      <property xmlns:ns="urn:partner.soap.sforce.com" xmlns:sf="urn:sobject.partner.soap.sforce.com" name="LastName" expression="//ns:queryResponse/ns:result/ns:records/sf:LastName/text()" scope="default" type="STRING"/>  
      <property xmlns:ns="urn:partner.soap.sforce.com" xmlns:sf="urn:sobject.partner.soap.sforce.com" name="Email" expression="//ns:queryResponse/ns:result/ns:records/sf:Email/text()" scope="default" type="STRING"/>  
      <log level="full" separator=","/>  
      <payloadFactory media-type="json">  
       <format>{ "ProfileId": { "source": "SALESFORCE", "id": "$1" },  "Name": "$2", "Username": "$3", "LastName":  "$4", "Email": "$5"  }</format>  
       <args>  
         <arg evaluator="xml" expression="$ctx:ProfileId"/>  
         <arg evaluator="xml" expression="$ctx:Name"/>  
         <arg evaluator="xml" expression="$ctx:Username"/>  
         <arg evaluator="xml" expression="$ctx:LastName"/>  
         <arg evaluator="xml" expression="$ctx:Email"/>  
       </args>  
      </payloadFactory>  
      <property name="messageType" value="application/json" scope="axis2" type="STRING"/>  
      <respond/>  
    </inSequence>  
   </resource>  
 </api>  


 

Thursday, July 9, 2015

Enable email login in WSO2 carbon products

To enable email address the below steps can be followed in any carbon product.

1. EnableEmailUserName in carbon.xml

<EnableEmailUserName>true</EnableEmailUserName>

2. Then provide the correct regex to allow email address in user store configuration in user-mgt.xml for JDBC user store
E.g.,

    <Property name="UsernameJavaRegEx">[a-
zA-Z0-9@._-|//]{3,30}$</Property>
3. Create admin user with email address in user in user-mgt.xml.

   <AdminUser>
                     <UserName>admin@wso2.com</UserName>
                     <Password>admin</Password>
  </AdminUser>
By the above configurations, it will enable email address.

If you want to give the both support, email address and username, you can include the below property in user store configuration.

4.  <Property name="
UsernameWithEmailJavaScriptRegEx">[a-zA-Z0-9@._-|//]{3,30}$</Property>
      
To know how to do this for a LDAP, refer this well explained blog post [1] done for Identity server which is applicable for other carbon products as well. This document also explains the properties [2]

Tuesday, July 7, 2015

A reason for getting com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Sometimes you will end up in getting an exception like below when starting a server pointed to MySQL DB.

ERROR - DatabaseUtil Database Error - Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server



One possible reason can be since you have mapped, bind-address to an IP in /etc/mysql/my.cnf

The default is 0.0.0.0 which is all interfaces. This setting does not restrict which IPs can access the server, unless you specified 127.0.0.1 for localhost only or some other IP.

bind-address            = 0.0.0.0

Once this is done, restart mysql server.

 sudo service mysql restart

Monday, May 11, 2015

Simple explaination for URL configurations done during IS as Key Manager in WSO2 API Manager

When you configure as IS as Key Manager, this document can be refereed. The below explanations for some of the URLs given in the configuration.

In WSO2 IS Side :


Make the following changes in the api-manager.xml file you just copied.

  • Change the <RevokeAPIURL> so that it points to the API Manager server. Note that if API Manager is running in distributed mode (has a separate node for the Gateway), you need to point this URL to the Gateway node. This is done so that when the token is revoked, the Gateway cache is updated as well. The port value you enter here must be the NIO port. See Default Ports of WSO2 Products for more information.
 <RevokeAPIURL>https://${GATEWAY_SERVER_HOST}:{nio/passthrough port}/revoke</RevokeAPIURL>

Why do we point to API Manager ? This is because this will call the Gateway node in API manager which points to the Key Manager

Why we point to the NIO/Passthru port ? This is because, here we will call the _RevokeAPI_.xml which is deployed in synapse folder of the Gateway. (In a distributed scenario, we point to the _RevokeAPI_.xml that resides in the Gateway worker node)

Note : So in a distributed scenario (APIM distributed scenario), In store if you call Revoke or Regenerate, store will call the key Manager/Validator and key Manager/Validator will call the gateway.

So in Store we have to configure KeyValidator/Key manager server url and in Key Manager/Validator we have to configure gateway server url (Passthru). Inside Gateway the apis will be called (NIO)


  • Change the <ServerURL> occurring under the <APIGateway> (of the Key Manager/Key Validator node) section so that it points to the API Manager server. If you are using distributed mode, this needs to point to the Gateway node as well. This is done so that when the token is regenerated, the Gateway cache is updated as well. The port value you enter here must be the management transport port.

<ServerURL>https://${GATEWAY_SERVER_HOST}:{port}/services/</ServerURL>

Why we point to API Manager ? This  is because this will call the Gateway node in API manager. This is to identify the gateway node.

Why we point to servlet port ? This is because it calls the admin services.

Note : This is same like as above explained. Since IS is the Key Validator/Manager it calls the gateway.


In WSO2 APIM Side :

Open the api-manager.xml file found in the <APIM_HOME>/repository/conf directory and change the following. 

  • Change the ServerURL of the AuthManager to point to IS.
  <ServerURL>https://${IS_SERVER_HOST}:{port}/services/</ServerURL>

Why IS : This is because authentication will be done via Key manager/validator

Why we point to servlet port ?
This is because it calls the admin services.

  • Change the ServerURL of the APIKeyManager to point to IS.
    <ServerURL>https://${IS_SERVER_HOST}:{port}/services/</ServerURL>

Why IS : This is because authentication will be done via Key manager/validator
Why we point to servlet ports : This is because it calls admin services

Usage of ports in a gateway cluster when fronted by Nginx:

GW manager : Use to publish APIs - So only the servlet ports will be used
GW worker : Use when invoking - So only the passthrough/NIO ports will be used

Note : When you configure API endpoints in gateway (Which is in synapse folder  (E.g.,     _AuthorizeAPI_.xml, _RevokeAPI_.xml, _TokenAPI_.xml), you should edit them only in gateway manager since it will depsync to workers. Otherwise it can cause issues.