Internet Computer Wiki - User contributions [en]

Proof of Personhood

2022-12-13T18:02:40Z

Satya.vusirikala: /* Call State at Various Stages */

= This page is still work in progress =

== Proof of Personhood ==
Decentralized protocols generally rely on voting to make decisions. Anyone can create an identity and cast their vote on a proposal. A naïve implementation of this leads to a sybil attack, where one attacker creates many identities and casts many votes in his favor.

Proof of work and proof of stake are the most commonly implemented sybil attack resistance mechanisms. In proof of work, each identity has to invest some computational power to cast votes. The identity's voting power is proportional to its computational power. In proof of stake, each identity has to purchase some tokens and stake it in the protocol. The identity's voting power is proportional to its stake. In both cases, people with more money gets higher voting power and can alter the decisions of the decentralized protocol.

Proof of personhood is an alternate solution to solve the sybil attack problem. The goal of the system is to make sure each real person gets voting power of (close to) only 1. Even if the person creates many identities, he cannot have a higher voting power. In this system, anyone create an identity but its voting power is 0 by default. After the identity proves that it is controlled by a real person, it is assigned a non-zero voting power. There are many well-known techniques such as CAPTCHA to verify that someone is a real person. However, a motivated person could still manually create many identities, manually prove each of them is controlled by a real person and get a high voting power. To protect against this attack, the system needs to satisfy a second requirement -- If a single person controls many identities, the sum of voting power of all the identities is still (close to) 1. Essentially, an attacker doesn't gain any advantage by creating many identities and proving each of them is controlled by a real person. The second requirement is quite difficult to satisfy.

This article describes the ''people parties'' project currently under development at Dfinity Foundation.

== People Parties ==
People parties<ref>https://medium.com/dfinity/ultimate-decentralization-using-virtual-people-parties-that-deliver-proof-of-personhood-at-de575522c80</ref> are a mechanism for obtaining proof of personhood (also referred to as proof of humanity) built on and for the Internet Computer.

Each people party takes place at one specific point in time. Prior to that time, each prospective participant commits to a location that they will visit for the time of the party. At the beginning of the party, participants are assigned to small, random subgroups, and meet in a real-time video call with other participants assigned to the same group. The video call, however, does not show the participants’ faces; instead, participants show their surroundings, proving that they are at the place they committed to. As locations chosen by different participants must have a certain minimum distance, and no participant can be physically present at two locations simultaneously, the proof of personhood even guarantees the uniqueness of the validated persons.

The main purpose of people parties is democratization. Each validated participant can designate a neuron in the [[Network Nervous System]] that receives increased voting power; this improves the relative voting power of the “many” vs. the “heavy”. It also provides additional voting rewards to all validated participants, which further motivates people to participate in the parties. The validated personhood also benefits the Internet Computer ecosystem more broadly: Open Internet Systems, which are dapps that are controlled by a decentralized governance system, will be able to profit from the improved decentralization similarly to the Internet Computer itself. And any dapp will be able to use the validation information in order to, e.g., differentiate between bots and actual human users.

== Architecture ==
The architecture of the people parties project involves 3 components.
* Client
* People party canister
* Signaling server

===== Client =====
The client is a device that is used by the participants who want to validate their identity. The client can be a desktop, laptop or a mobile device with a web browser.

===== People party canister =====
People party canister is a canister (software program) hosted on the Internet Computer. The canister maintains the information related to all the people parties and identities. This includes
* The longitude boundaries of the participants allowed for the party
* List of participants in the party
* The state of each call in the party
* Validation results of each participant
* Personhood score (voting power) of each identity

The client queries people party canister to obtain the list of all the upcoming parties. The client can then register or deregister for any party by calling the people party canister. During the registration, the client specifies the latitude and longitude coordinates of the location he will be during the party.

A few minutes before a party commences, the client calls people party canister to join the party. Throughout the party, the client repeatedly calls the people party canister to obtain the state of the call. During the call, each participant streams a video that they are in the promised location. Other participants verify this by simultaneously glancing at the google street view of the location, and send an approve/decline vote to the people party canister. After the call ends, the client queries the people party canister to know if the client was approved/declined by others. The client later calls the people party canister to withdraw 1 ICP it deposited to register for the party.

===== Signaling server =====
Signaling server is used to facilitate video calls amongst the participants.

== People Party Canister Interface ==
We describe the interface supported by the people party canister here.

get_parties: () -> (vec record { PartyId; PartySpecification; }) query;
register: (id: PartyId, place: Location) -> (RegistrationResponse);
deregister: (id: PartyId) -> (DeregistrationResponse);
withdraw: (destination: AccountIdentifier) -> (WithdrawResult);
join: (id: PartyId, key: blob) -> (JoinResponse);
vote: (id: PartyId, vote: record { Vote; ParticipantName; }) -> (VoteResponse);
http_request: (request: HttpRequest) -> (HttpResponse) query;
get_person_profile: () -> (opt PersonProfile) query;
get_call_state: (id: PartyId) -> (PublicCallState) query;
get_call_result: (id: PartyId) -> (CallResult) query;

When a user enters the people party website, the frontend calls <code>get_parties</code> endpoint to display the information of all the parties. The <code>PartySpecification</code> for each people party is described as follows.
type PartySpecification =
record {
registration_start: Time;
registration_end: Time;
call_start : Time;
longitude_min : float64;
longitude_max : float64;
};

The user can then then register or deregister for a party by calling <code>register</code> and <code>deregister</code> endpoints. The canister responds with the status of registration. When the user registers for a party, he specifies the the Location (latitude and longitude) of where the user will be when the party starts.

A few minutes before the party starts, the user joins the party by calling <code>join</code> endpoint. After the user joins a call, the user sends one vote (approve/decline) for each of the other participants by calling <code>vote</code> endpoint.

At any point in time, one can call <code>get_person_profile</code>, <code>get_call_state</code> and <code>get_call_result</code> endpoints to obtain the information about the list of parties, a person's profile, state of a call and result of a call respectively.

The profile of a person is described by
type PersonProfile = record {
upcoming_parties: vec record { PartyId; Location; };
validation_score: nat64;
past_parties: vec PartyId;
};

When the user registers for a party, he needs to deposit 1 ICP. After the party ends, the user can withdraw this amount by calling <code>withdraw</code> endpoint. The user can only withdraw if he hasn't registered for any upcoming party. Once the user withdraws the amount, he cannot register again until he deposits 1 ICP.

== Party Flow ==
The below flow describing a call. Say the call is scheduled to start at 10am. Then the timeline should be as follows:

* Until 9:55am, users can enroll with the ''register'' endpoint. Since the assignment to groups has not been made, calling <code>get_call_state</code> will return what is in case 0 below.
* At 9:55am, the canister closes the registration phase and calling the <code>register</code> endpoint for the current party is not allowed anymore.
* From 9:55am to 10am, users are allowed to join. For this, there is a join endpoint that accepts a key – a blob. Calling <code>get_call_state</code> will return the result described in case 1 below.
* At 10am, the canister generates the random setup of all calls. From this time onward, calling the join endpoint is not allowed anymore, and clients are supposed to set up the WebRTC connections.
* Between 10am and 10:01am, the canister returns the call setup state as described in step 1.5 below.
* At 10:01am, the actual calls start.
* From 10:01am to 10:11am, the actual party is supposed to take place and users can submit votes. Calling <code>get_call_state</code> will return the result described in case 2 below.
* At 10:11am, the canister closes all calls and makes the tally. From this point onward, calling vote is not allowed anymore.
* From 10:11am, <code>get_call_state</code> will return the result described in case 3 below.

== Call State at Various Stages ==
The people parties canister stores the state of each call, and updates the state during each phase of the party. The canister exposes the <code>get_call_state</code> method that returns the state of the call.

==== Before Registration ====
This is returned for parties that do not exist:
call_state = record {
public_call_state = variant{not_created};
my_votes = vec{};
};

==== Joining – Before call starts ====
This is returned from the point in time where the groups are assigned but before the point in time when the call is scheduled to start. The state transition from 1 to 1.5 is defined by the canister computing the random assignment.
call_state = record {
public_call_state = variant{not_started = record {
joined = true; ← so the UI can show that the user is in the waiting room
starts_in_seconds = 30;
}};
my_votes = vec{};
};

==== Starting the call ====
call_state = record {
public_call_state = variant{starting = record{
myself = “adhesive bread”;
participants = vec{
record{
name = “adhesive bread”;
location = …
key = [30, 57, 30, …];
};
record{
Name = “stinky tofu”;
Location = …
key = [30, 57, 30, …];
...
};
starts_in_seconds = 30;
}};
my_votes = vec{};
};

==== After call starts – before call ends ====
If the user hasn’t successfully joined the call, then the following is returned:
call_state = record {
public_call_state = variant{not_joined};
};

If the user has successfully joined the call, then the following is returned:
call_state = record {
public_call_state = variant{active = record{
myself = “adhesive bread”;
participants = vec{
record{
name = “adhesive bread”;
location = …
key = [30, 57, 30, …];
};
...
};
round = 0;
remaining_seconds = 42;
voters_in_round = vec{“chilly mustard”; “angry fox”;};
}}
};
State in later round:
call_state = record {
public_call_state = variant{active = record{
myself = “adhesive bread”;
participants = vec{ ... };
round = 1;
remaining_seconds = 22;
voters_in_round = vec{“adhesive bread”; “angry fox”;};
}}
};

==== After call ends ====
call_state = record {
public_call_state = variant{ended};
};

== Personhood Score ==
Each identity in the Internet Computer is assigned a personhood score. If an identity hasn’t participated in any people party, its personhood score is 0. As the identity participates in people parties and is successfully validated to be a person, its personhood score increases.

Any canister will be able to query and obtain the personhood score of an identity. One can potentially use the personhood score to determine the voting power of an identity in an election/voting program.

===== Requirement #1 =====
An Internet Computer user is naturally inclined to increase his voting power. He is therefore motivated to create many identities and attend a people party with each identity. This gives him control of many identities with a positive personhood score, and thereby a higher voting power. We would like to avoid this. We would like to design the personhood score in such a way that an IC user is motivated to attend each people party with the same identity rather than creating a new identity for each people party.

===== Requirement #2 =====
Suppose Alice attended many parties with her identity, and Bob attended only one recent party. Alice and Bob’s personhood scores should still be close. Alice should not have a significant advantage by attending many parties.

===== Failed Attempt #1 =====
Suppose personhood score of an identity = 1 if the identity is successfully validated in at least one party. This personhood score violates our Requirement #1.

===== Failed Attempt #2 =====
Suppose personhood score of an identity = number of parties in which the identity is successfully validated. This personhood score violates our Requirement #2.

===== Successful Attempt =====
After a party ends, the personhood score of an identity is composed of three components:
* Recent party result (RPR): This value is 1 if the identity is accepted to be a person in the recent party. The value is 0 if the identity did not participate or is rejected in the recent party.
* Decay function of previous personhood score: The decay function is a monotonically non-increasing function. It is computed on the old personhood score (the score before the recent party started). The decay lets us give more weightage to the validation in the recent party over the previous parties.
* Bonus reward function: If the identity is accepted in the most recent party and is successfully accepted in some parties before, a bonus reward is given to the identity. This bonus component is added to incentivize IC users to participate in all the parties using the same identity rather than creating a different identity for each party.

Suppose the personhood score of an identity is PSold before a party starts. The trust score PSnew of the identity after the party ends is calculated as follows.

PSnew = RPR * (1 + bonus(PSold)) + decay(PSold)

Where Recent Party Result (RPR) is 1 iff the identity is accepted in the party.

Through empirical evidence, we found that
* bonus(x) = log10(1 + 0.25 * x) and
* decay(x) = log10(1 + x)
work pretty well. That is,

PSnew = RPR * (1 + log10(1 + 0.25 * PSold)) + log10(1 + PSold)

===== Example =====
Consider 2 scenarios.
* Scenario A: The user enters each party with the same identity, and the user is accepted in each of the parties.
* Scenario B: The user enters each party with a new identity, and the user is accepted in each of the parties.

Let us compare the personhood scores of the user in each of the scenarios. In Scenario B, we sum up the personhood scores of all the identities controlled by the user.

{| class="wikitable"
|+ Comparison of personhood scores of a user in Scenarios A & B
|-
! !! Scenario A !! Scenario B
|-
| Initial Score || 0 || 0
|-
| After 1st party || 1 || 1
|-
| After 2nd party || 1.398 || 1.301
|-
| After 3rd party || 1.510 || 1.415
|-
| After 4th party || 1.539 || 1.462
|-
| After 5th party || 1.546 || 1.482
|-
| After 6th party || 1.548 || 1.491
|-
| After 7th party || 1.548 || 1.494
|-
| After 8th party || 1.548 || 1.496
|}

Proof of Personhood

2022-12-13T18:01:05Z

Satya.vusirikala: /* Call State at Various Stages */

= This page is still work in progress =

== Proof of Personhood ==
Decentralized protocols generally rely on voting to make decisions. Anyone can create an identity and cast their vote on a proposal. A naïve implementation of this leads to a sybil attack, where one attacker creates many identities and casts many votes in his favor.

Proof of work and proof of stake are the most commonly implemented sybil attack resistance mechanisms. In proof of work, each identity has to invest some computational power to cast votes. The identity's voting power is proportional to its computational power. In proof of stake, each identity has to purchase some tokens and stake it in the protocol. The identity's voting power is proportional to its stake. In both cases, people with more money gets higher voting power and can alter the decisions of the decentralized protocol.

Proof of personhood is an alternate solution to solve the sybil attack problem. The goal of the system is to make sure each real person gets voting power of (close to) only 1. Even if the person creates many identities, he cannot have a higher voting power. In this system, anyone create an identity but its voting power is 0 by default. After the identity proves that it is controlled by a real person, it is assigned a non-zero voting power. There are many well-known techniques such as CAPTCHA to verify that someone is a real person. However, a motivated person could still manually create many identities, manually prove each of them is controlled by a real person and get a high voting power. To protect against this attack, the system needs to satisfy a second requirement -- If a single person controls many identities, the sum of voting power of all the identities is still (close to) 1. Essentially, an attacker doesn't gain any advantage by creating many identities and proving each of them is controlled by a real person. The second requirement is quite difficult to satisfy.

This article describes the ''people parties'' project currently under development at Dfinity Foundation.

== People Parties ==
People parties<ref>https://medium.com/dfinity/ultimate-decentralization-using-virtual-people-parties-that-deliver-proof-of-personhood-at-de575522c80</ref> are a mechanism for obtaining proof of personhood (also referred to as proof of humanity) built on and for the Internet Computer.

Each people party takes place at one specific point in time. Prior to that time, each prospective participant commits to a location that they will visit for the time of the party. At the beginning of the party, participants are assigned to small, random subgroups, and meet in a real-time video call with other participants assigned to the same group. The video call, however, does not show the participants’ faces; instead, participants show their surroundings, proving that they are at the place they committed to. As locations chosen by different participants must have a certain minimum distance, and no participant can be physically present at two locations simultaneously, the proof of personhood even guarantees the uniqueness of the validated persons.

The main purpose of people parties is democratization. Each validated participant can designate a neuron in the [[Network Nervous System]] that receives increased voting power; this improves the relative voting power of the “many” vs. the “heavy”. It also provides additional voting rewards to all validated participants, which further motivates people to participate in the parties. The validated personhood also benefits the Internet Computer ecosystem more broadly: Open Internet Systems, which are dapps that are controlled by a decentralized governance system, will be able to profit from the improved decentralization similarly to the Internet Computer itself. And any dapp will be able to use the validation information in order to, e.g., differentiate between bots and actual human users.

== Architecture ==
The architecture of the people parties project involves 3 components.
* Client
* People party canister
* Signaling server

===== Client =====
The client is a device that is used by the participants who want to validate their identity. The client can be a desktop, laptop or a mobile device with a web browser.

===== People party canister =====
People party canister is a canister (software program) hosted on the Internet Computer. The canister maintains the information related to all the people parties and identities. This includes
* The longitude boundaries of the participants allowed for the party
* List of participants in the party
* The state of each call in the party
* Validation results of each participant
* Personhood score (voting power) of each identity

The client queries people party canister to obtain the list of all the upcoming parties. The client can then register or deregister for any party by calling the people party canister. During the registration, the client specifies the latitude and longitude coordinates of the location he will be during the party.

A few minutes before a party commences, the client calls people party canister to join the party. Throughout the party, the client repeatedly calls the people party canister to obtain the state of the call. During the call, each participant streams a video that they are in the promised location. Other participants verify this by simultaneously glancing at the google street view of the location, and send an approve/decline vote to the people party canister. After the call ends, the client queries the people party canister to know if the client was approved/declined by others. The client later calls the people party canister to withdraw 1 ICP it deposited to register for the party.

===== Signaling server =====
Signaling server is used to facilitate video calls amongst the participants.

== People Party Canister Interface ==
We describe the interface supported by the people party canister here.

get_parties: () -> (vec record { PartyId; PartySpecification; }) query;
register: (id: PartyId, place: Location) -> (RegistrationResponse);
deregister: (id: PartyId) -> (DeregistrationResponse);
withdraw: (destination: AccountIdentifier) -> (WithdrawResult);
join: (id: PartyId, key: blob) -> (JoinResponse);
vote: (id: PartyId, vote: record { Vote; ParticipantName; }) -> (VoteResponse);
http_request: (request: HttpRequest) -> (HttpResponse) query;
get_person_profile: () -> (opt PersonProfile) query;
get_call_state: (id: PartyId) -> (PublicCallState) query;
get_call_result: (id: PartyId) -> (CallResult) query;

When a user enters the people party website, the frontend calls <code>get_parties</code> endpoint to display the information of all the parties. The <code>PartySpecification</code> for each people party is described as follows.
type PartySpecification =
record {
registration_start: Time;
registration_end: Time;
call_start : Time;
longitude_min : float64;
longitude_max : float64;
};

The user can then then register or deregister for a party by calling <code>register</code> and <code>deregister</code> endpoints. The canister responds with the status of registration. When the user registers for a party, he specifies the the Location (latitude and longitude) of where the user will be when the party starts.

A few minutes before the party starts, the user joins the party by calling <code>join</code> endpoint. After the user joins a call, the user sends one vote (approve/decline) for each of the other participants by calling <code>vote</code> endpoint.

At any point in time, one can call <code>get_person_profile</code>, <code>get_call_state</code> and <code>get_call_result</code> endpoints to obtain the information about the list of parties, a person's profile, state of a call and result of a call respectively.

The profile of a person is described by
type PersonProfile = record {
upcoming_parties: vec record { PartyId; Location; };
validation_score: nat64;
past_parties: vec PartyId;
};

When the user registers for a party, he needs to deposit 1 ICP. After the party ends, the user can withdraw this amount by calling <code>withdraw</code> endpoint. The user can only withdraw if he hasn't registered for any upcoming party. Once the user withdraws the amount, he cannot register again until he deposits 1 ICP.

== Party Flow ==
The below flow describing a call. Say the call is scheduled to start at 10am. Then the timeline should be as follows:

* Until 9:55am, users can enroll with the ''register'' endpoint. Since the assignment to groups has not been made, calling <code>get_call_state</code> will return what is in case 0 below.
* At 9:55am, the canister closes the registration phase and calling the <code>register</code> endpoint for the current party is not allowed anymore.
* From 9:55am to 10am, users are allowed to join. For this, there is a join endpoint that accepts a key – a blob. Calling <code>get_call_state</code> will return the result described in case 1 below.
* At 10am, the canister generates the random setup of all calls. From this time onward, calling the join endpoint is not allowed anymore, and clients are supposed to set up the WebRTC connections.
* Between 10am and 10:01am, the canister returns the call setup state as described in step 1.5 below.
* At 10:01am, the actual calls start.
* From 10:01am to 10:11am, the actual party is supposed to take place and users can submit votes. Calling <code>get_call_state</code> will return the result described in case 2 below.
* At 10:11am, the canister closes all calls and makes the tally. From this point onward, calling vote is not allowed anymore.
* From 10:11am, <code>get_call_state</code> will return the result described in case 3 below.

== Call State at Various Stages ==
The people parties canister stores the state of each call, and updates the state during each phase of the party. The canister exposes the <code>get_call_state</code> method that returns the state of the call.

==== Before Registration ====
This is returned for parties that do not exist:
call_state = record {
public_call_state = variant{not_created};
my_votes = vec{};
};

==== Joining – Before call starts ====
This is returned from the point in time where the groups are assigned but before the point in time when the call is scheduled to start. The state transition from 1 to 1.5 is defined by the canister computing the random assignment.
call_state = record {
public_call_state = variant{not_started = record {
joined = true; ← so the UI can show that the user is in the waiting room
starts_in_seconds = 30;
}};
my_votes = vec{};
};

==== Starting the call ====
call_state = record {
public_call_state = variant{starting = record{
myself = “adhesive bread”;
participants = vec{
record{
name = “adhesive bread”;
location = …
key = [30, 57, 30, …];
};
record{
Name = “stinky tofu”;
Location = …
key = [30, 57, 30, …];
...
};
starts_in_seconds = 30;
}};
my_votes = vec{};
};

==== After call starts – before call ends ====
If the user hasn’t successfully joined the call, then the following is returned:
call_state = record {
public_call_state = variant{not_joined};
};

If the user has successfully joined the call, then the following is returned:
call_state = record {
public_call_state = variant{active = record{
myself = “adhesive bread”;
participants = vec{
record{
name = “adhesive bread”;
location = …
key = [30, 57, 30, …];
};
...
};
round = 0;
remaining_seconds = 42;
voters_in_round = vec{“chilly mustard”; “angry fox”;};
}}
};

==== After call ends ====
call_state = record {
public_call_state = variant{ended};
};

== Personhood Score ==
Each identity in the Internet Computer is assigned a personhood score. If an identity hasn’t participated in any people party, its personhood score is 0. As the identity participates in people parties and is successfully validated to be a person, its personhood score increases.

Any canister will be able to query and obtain the personhood score of an identity. One can potentially use the personhood score to determine the voting power of an identity in an election/voting program.

===== Requirement #1 =====
An Internet Computer user is naturally inclined to increase his voting power. He is therefore motivated to create many identities and attend a people party with each identity. This gives him control of many identities with a positive personhood score, and thereby a higher voting power. We would like to avoid this. We would like to design the personhood score in such a way that an IC user is motivated to attend each people party with the same identity rather than creating a new identity for each people party.

===== Requirement #2 =====
Suppose Alice attended many parties with her identity, and Bob attended only one recent party. Alice and Bob’s personhood scores should still be close. Alice should not have a significant advantage by attending many parties.

===== Failed Attempt #1 =====
Suppose personhood score of an identity = 1 if the identity is successfully validated in at least one party. This personhood score violates our Requirement #1.

===== Failed Attempt #2 =====
Suppose personhood score of an identity = number of parties in which the identity is successfully validated. This personhood score violates our Requirement #2.

===== Successful Attempt =====
After a party ends, the personhood score of an identity is composed of three components:
* Recent party result (RPR): This value is 1 if the identity is accepted to be a person in the recent party. The value is 0 if the identity did not participate or is rejected in the recent party.
* Decay function of previous personhood score: The decay function is a monotonically non-increasing function. It is computed on the old personhood score (the score before the recent party started). The decay lets us give more weightage to the validation in the recent party over the previous parties.
* Bonus reward function: If the identity is accepted in the most recent party and is successfully accepted in some parties before, a bonus reward is given to the identity. This bonus component is added to incentivize IC users to participate in all the parties using the same identity rather than creating a different identity for each party.

Suppose the personhood score of an identity is PSold before a party starts. The trust score PSnew of the identity after the party ends is calculated as follows.

PSnew = RPR * (1 + bonus(PSold)) + decay(PSold)

Where Recent Party Result (RPR) is 1 iff the identity is accepted in the party.

Through empirical evidence, we found that
* bonus(x) = log10(1 + 0.25 * x) and
* decay(x) = log10(1 + x)
work pretty well. That is,

PSnew = RPR * (1 + log10(1 + 0.25 * PSold)) + log10(1 + PSold)

===== Example =====
Consider 2 scenarios.
* Scenario A: The user enters each party with the same identity, and the user is accepted in each of the parties.
* Scenario B: The user enters each party with a new identity, and the user is accepted in each of the parties.

Let us compare the personhood scores of the user in each of the scenarios. In Scenario B, we sum up the personhood scores of all the identities controlled by the user.

{| class="wikitable"
|+ Comparison of personhood scores of a user in Scenarios A & B
|-
! !! Scenario A !! Scenario B
|-
| Initial Score || 0 || 0
|-
| After 1st party || 1 || 1
|-
| After 2nd party || 1.398 || 1.301
|-
| After 3rd party || 1.510 || 1.415
|-
| After 4th party || 1.539 || 1.462
|-
| After 5th party || 1.546 || 1.482
|-
| After 6th party || 1.548 || 1.491
|-
| After 7th party || 1.548 || 1.494
|-
| After 8th party || 1.548 || 1.496
|}

Proof of Personhood

2022-12-13T18:00:25Z

Satya.vusirikala: /* Call State at Various Stages */

= This page is still work in progress =

== Proof of Personhood ==
Decentralized protocols generally rely on voting to make decisions. Anyone can create an identity and cast their vote on a proposal. A naïve implementation of this leads to a sybil attack, where one attacker creates many identities and casts many votes in his favor.

Proof of work and proof of stake are the most commonly implemented sybil attack resistance mechanisms. In proof of work, each identity has to invest some computational power to cast votes. The identity's voting power is proportional to its computational power. In proof of stake, each identity has to purchase some tokens and stake it in the protocol. The identity's voting power is proportional to its stake. In both cases, people with more money gets higher voting power and can alter the decisions of the decentralized protocol.

Proof of personhood is an alternate solution to solve the sybil attack problem. The goal of the system is to make sure each real person gets voting power of (close to) only 1. Even if the person creates many identities, he cannot have a higher voting power. In this system, anyone create an identity but its voting power is 0 by default. After the identity proves that it is controlled by a real person, it is assigned a non-zero voting power. There are many well-known techniques such as CAPTCHA to verify that someone is a real person. However, a motivated person could still manually create many identities, manually prove each of them is controlled by a real person and get a high voting power. To protect against this attack, the system needs to satisfy a second requirement -- If a single person controls many identities, the sum of voting power of all the identities is still (close to) 1. Essentially, an attacker doesn't gain any advantage by creating many identities and proving each of them is controlled by a real person. The second requirement is quite difficult to satisfy.

This article describes the ''people parties'' project currently under development at Dfinity Foundation.

== People Parties ==
People parties<ref>https://medium.com/dfinity/ultimate-decentralization-using-virtual-people-parties-that-deliver-proof-of-personhood-at-de575522c80</ref> are a mechanism for obtaining proof of personhood (also referred to as proof of humanity) built on and for the Internet Computer.

Each people party takes place at one specific point in time. Prior to that time, each prospective participant commits to a location that they will visit for the time of the party. At the beginning of the party, participants are assigned to small, random subgroups, and meet in a real-time video call with other participants assigned to the same group. The video call, however, does not show the participants’ faces; instead, participants show their surroundings, proving that they are at the place they committed to. As locations chosen by different participants must have a certain minimum distance, and no participant can be physically present at two locations simultaneously, the proof of personhood even guarantees the uniqueness of the validated persons.

The main purpose of people parties is democratization. Each validated participant can designate a neuron in the [[Network Nervous System]] that receives increased voting power; this improves the relative voting power of the “many” vs. the “heavy”. It also provides additional voting rewards to all validated participants, which further motivates people to participate in the parties. The validated personhood also benefits the Internet Computer ecosystem more broadly: Open Internet Systems, which are dapps that are controlled by a decentralized governance system, will be able to profit from the improved decentralization similarly to the Internet Computer itself. And any dapp will be able to use the validation information in order to, e.g., differentiate between bots and actual human users.

== Architecture ==
The architecture of the people parties project involves 3 components.
* Client
* People party canister
* Signaling server

===== Client =====
The client is a device that is used by the participants who want to validate their identity. The client can be a desktop, laptop or a mobile device with a web browser.

===== People party canister =====
People party canister is a canister (software program) hosted on the Internet Computer. The canister maintains the information related to all the people parties and identities. This includes
* The longitude boundaries of the participants allowed for the party
* List of participants in the party
* The state of each call in the party
* Validation results of each participant
* Personhood score (voting power) of each identity

The client queries people party canister to obtain the list of all the upcoming parties. The client can then register or deregister for any party by calling the people party canister. During the registration, the client specifies the latitude and longitude coordinates of the location he will be during the party.

A few minutes before a party commences, the client calls people party canister to join the party. Throughout the party, the client repeatedly calls the people party canister to obtain the state of the call. During the call, each participant streams a video that they are in the promised location. Other participants verify this by simultaneously glancing at the google street view of the location, and send an approve/decline vote to the people party canister. After the call ends, the client queries the people party canister to know if the client was approved/declined by others. The client later calls the people party canister to withdraw 1 ICP it deposited to register for the party.

===== Signaling server =====
Signaling server is used to facilitate video calls amongst the participants.

== People Party Canister Interface ==
We describe the interface supported by the people party canister here.

get_parties: () -> (vec record { PartyId; PartySpecification; }) query;
register: (id: PartyId, place: Location) -> (RegistrationResponse);
deregister: (id: PartyId) -> (DeregistrationResponse);
withdraw: (destination: AccountIdentifier) -> (WithdrawResult);
join: (id: PartyId, key: blob) -> (JoinResponse);
vote: (id: PartyId, vote: record { Vote; ParticipantName; }) -> (VoteResponse);
http_request: (request: HttpRequest) -> (HttpResponse) query;
get_person_profile: () -> (opt PersonProfile) query;
get_call_state: (id: PartyId) -> (PublicCallState) query;
get_call_result: (id: PartyId) -> (CallResult) query;

When a user enters the people party website, the frontend calls <code>get_parties</code> endpoint to display the information of all the parties. The <code>PartySpecification</code> for each people party is described as follows.
type PartySpecification =
record {
registration_start: Time;
registration_end: Time;
call_start : Time;
longitude_min : float64;
longitude_max : float64;
};

The user can then then register or deregister for a party by calling <code>register</code> and <code>deregister</code> endpoints. The canister responds with the status of registration. When the user registers for a party, he specifies the the Location (latitude and longitude) of where the user will be when the party starts.

A few minutes before the party starts, the user joins the party by calling <code>join</code> endpoint. After the user joins a call, the user sends one vote (approve/decline) for each of the other participants by calling <code>vote</code> endpoint.

At any point in time, one can call <code>get_person_profile</code>, <code>get_call_state</code> and <code>get_call_result</code> endpoints to obtain the information about the list of parties, a person's profile, state of a call and result of a call respectively.

The profile of a person is described by
type PersonProfile = record {
upcoming_parties: vec record { PartyId; Location; };
validation_score: nat64;
past_parties: vec PartyId;
};

When the user registers for a party, he needs to deposit 1 ICP. After the party ends, the user can withdraw this amount by calling <code>withdraw</code> endpoint. The user can only withdraw if he hasn't registered for any upcoming party. Once the user withdraws the amount, he cannot register again until he deposits 1 ICP.

== Party Flow ==
The below flow describing a call. Say the call is scheduled to start at 10am. Then the timeline should be as follows:

* Until 9:55am, users can enroll with the ''register'' endpoint. Since the assignment to groups has not been made, calling <code>get_call_state</code> will return what is in case 0 below.
* At 9:55am, the canister closes the registration phase and calling the <code>register</code> endpoint for the current party is not allowed anymore.
* From 9:55am to 10am, users are allowed to join. For this, there is a join endpoint that accepts a key – a blob. Calling <code>get_call_state</code> will return the result described in case 1 below.
* At 10am, the canister generates the random setup of all calls. From this time onward, calling the join endpoint is not allowed anymore, and clients are supposed to set up the WebRTC connections.
* Between 10am and 10:01am, the canister returns the call setup state as described in step 1.5 below.
* At 10:01am, the actual calls start.
* From 10:01am to 10:11am, the actual party is supposed to take place and users can submit votes. Calling <code>get_call_state</code> will return the result described in case 2 below.
* At 10:11am, the canister closes all calls and makes the tally. From this point onward, calling vote is not allowed anymore.
* From 10:11am, <code>get_call_state</code> will return the result described in case 3 below.

== Call State at Various Stages ==
The people parties canister stores the state of each call, and updates the state during each phase of the party. The canister exposes the <code>get_call_state</code> method that returns the state of the call.

==== Before Registration ====
This is returned for parties that do not exist:
call_state = record {
public_call_state = variant{not_created};
my_votes = vec{};
};

==== Joining – Before call starts ====
This is returned from the point in time where the groups are assigned but before the point in time when the call is scheduled to start. The state transition from 1 to 1.5 is defined by the canister computing the random assignment.
call_state = record {
public_call_state = variant{not_started = record {
joined = true; ← so the UI can show that the user is in the waiting room
starts_in_seconds = 30;
}};
my_votes = vec{};
};

==== Starting the call ====
call_state = record {
public_call_state = variant{starting = record{
myself = “adhesive bread”;
participants = vec{
record{
name = “adhesive bread”;
location = …
key = [30, 57, 30, …];
};
record{
Name = “stinky tofu”;
Location = …
key = [30, 57, 30, …];
...
};
starts_in_seconds = 30;
}};
my_votes = vec{};
};

==== After call starts – before call ends ====
If the user hasn’t successfully joined the call, then the following is returned:
call_state = record {
public_call_state = variant{not_joined};
};

If the user has successfully joined the call, then the following is returned:
call_state = record {
public_call_state = variant{active = record{
myself = “adhesive bread”;
participants = vec{
record{
name = “adhesive bread”;
location = …
key = [30, 57, 30, …];
};
...
};
round = 0;
remaining_seconds = 42;
voters_in_round = vec{“chilly mustard”; “angry fox”;};
}}
};

== Personhood Score ==
Each identity in the Internet Computer is assigned a personhood score. If an identity hasn’t participated in any people party, its personhood score is 0. As the identity participates in people parties and is successfully validated to be a person, its personhood score increases.

Any canister will be able to query and obtain the personhood score of an identity. One can potentially use the personhood score to determine the voting power of an identity in an election/voting program.

===== Requirement #1 =====
An Internet Computer user is naturally inclined to increase his voting power. He is therefore motivated to create many identities and attend a people party with each identity. This gives him control of many identities with a positive personhood score, and thereby a higher voting power. We would like to avoid this. We would like to design the personhood score in such a way that an IC user is motivated to attend each people party with the same identity rather than creating a new identity for each people party.

===== Requirement #2 =====
Suppose Alice attended many parties with her identity, and Bob attended only one recent party. Alice and Bob’s personhood scores should still be close. Alice should not have a significant advantage by attending many parties.

===== Failed Attempt #1 =====
Suppose personhood score of an identity = 1 if the identity is successfully validated in at least one party. This personhood score violates our Requirement #1.

===== Failed Attempt #2 =====
Suppose personhood score of an identity = number of parties in which the identity is successfully validated. This personhood score violates our Requirement #2.

===== Successful Attempt =====
After a party ends, the personhood score of an identity is composed of three components:
* Recent party result (RPR): This value is 1 if the identity is accepted to be a person in the recent party. The value is 0 if the identity did not participate or is rejected in the recent party.
* Decay function of previous personhood score: The decay function is a monotonically non-increasing function. It is computed on the old personhood score (the score before the recent party started). The decay lets us give more weightage to the validation in the recent party over the previous parties.
* Bonus reward function: If the identity is accepted in the most recent party and is successfully accepted in some parties before, a bonus reward is given to the identity. This bonus component is added to incentivize IC users to participate in all the parties using the same identity rather than creating a different identity for each party.

Suppose the personhood score of an identity is PSold before a party starts. The trust score PSnew of the identity after the party ends is calculated as follows.

PSnew = RPR * (1 + bonus(PSold)) + decay(PSold)

Where Recent Party Result (RPR) is 1 iff the identity is accepted in the party.

Through empirical evidence, we found that
* bonus(x) = log10(1 + 0.25 * x) and
* decay(x) = log10(1 + x)
work pretty well. That is,

PSnew = RPR * (1 + log10(1 + 0.25 * PSold)) + log10(1 + PSold)

===== Example =====
Consider 2 scenarios.
* Scenario A: The user enters each party with the same identity, and the user is accepted in each of the parties.
* Scenario B: The user enters each party with a new identity, and the user is accepted in each of the parties.

Let us compare the personhood scores of the user in each of the scenarios. In Scenario B, we sum up the personhood scores of all the identities controlled by the user.

{| class="wikitable"
|+ Comparison of personhood scores of a user in Scenarios A & B
|-
! !! Scenario A !! Scenario B
|-
| Initial Score || 0 || 0
|-
| After 1st party || 1 || 1
|-
| After 2nd party || 1.398 || 1.301
|-
| After 3rd party || 1.510 || 1.415
|-
| After 4th party || 1.539 || 1.462
|-
| After 5th party || 1.546 || 1.482
|-
| After 6th party || 1.548 || 1.491
|-
| After 7th party || 1.548 || 1.494
|-
| After 8th party || 1.548 || 1.496
|}

Proof of Personhood

2022-12-13T17:59:13Z

Satya.vusirikala: /* Call State at Various Stages */

= This page is still work in progress =

== Proof of Personhood ==
Decentralized protocols generally rely on voting to make decisions. Anyone can create an identity and cast their vote on a proposal. A naïve implementation of this leads to a sybil attack, where one attacker creates many identities and casts many votes in his favor.

Proof of work and proof of stake are the most commonly implemented sybil attack resistance mechanisms. In proof of work, each identity has to invest some computational power to cast votes. The identity's voting power is proportional to its computational power. In proof of stake, each identity has to purchase some tokens and stake it in the protocol. The identity's voting power is proportional to its stake. In both cases, people with more money gets higher voting power and can alter the decisions of the decentralized protocol.

Proof of personhood is an alternate solution to solve the sybil attack problem. The goal of the system is to make sure each real person gets voting power of (close to) only 1. Even if the person creates many identities, he cannot have a higher voting power. In this system, anyone create an identity but its voting power is 0 by default. After the identity proves that it is controlled by a real person, it is assigned a non-zero voting power. There are many well-known techniques such as CAPTCHA to verify that someone is a real person. However, a motivated person could still manually create many identities, manually prove each of them is controlled by a real person and get a high voting power. To protect against this attack, the system needs to satisfy a second requirement -- If a single person controls many identities, the sum of voting power of all the identities is still (close to) 1. Essentially, an attacker doesn't gain any advantage by creating many identities and proving each of them is controlled by a real person. The second requirement is quite difficult to satisfy.

This article describes the ''people parties'' project currently under development at Dfinity Foundation.

== People Parties ==
People parties<ref>https://medium.com/dfinity/ultimate-decentralization-using-virtual-people-parties-that-deliver-proof-of-personhood-at-de575522c80</ref> are a mechanism for obtaining proof of personhood (also referred to as proof of humanity) built on and for the Internet Computer.

Each people party takes place at one specific point in time. Prior to that time, each prospective participant commits to a location that they will visit for the time of the party. At the beginning of the party, participants are assigned to small, random subgroups, and meet in a real-time video call with other participants assigned to the same group. The video call, however, does not show the participants’ faces; instead, participants show their surroundings, proving that they are at the place they committed to. As locations chosen by different participants must have a certain minimum distance, and no participant can be physically present at two locations simultaneously, the proof of personhood even guarantees the uniqueness of the validated persons.

The main purpose of people parties is democratization. Each validated participant can designate a neuron in the [[Network Nervous System]] that receives increased voting power; this improves the relative voting power of the “many” vs. the “heavy”. It also provides additional voting rewards to all validated participants, which further motivates people to participate in the parties. The validated personhood also benefits the Internet Computer ecosystem more broadly: Open Internet Systems, which are dapps that are controlled by a decentralized governance system, will be able to profit from the improved decentralization similarly to the Internet Computer itself. And any dapp will be able to use the validation information in order to, e.g., differentiate between bots and actual human users.

== Architecture ==
The architecture of the people parties project involves 3 components.
* Client
* People party canister
* Signaling server

===== Client =====
The client is a device that is used by the participants who want to validate their identity. The client can be a desktop, laptop or a mobile device with a web browser.

===== People party canister =====
People party canister is a canister (software program) hosted on the Internet Computer. The canister maintains the information related to all the people parties and identities. This includes
* The longitude boundaries of the participants allowed for the party
* List of participants in the party
* The state of each call in the party
* Validation results of each participant
* Personhood score (voting power) of each identity

The client queries people party canister to obtain the list of all the upcoming parties. The client can then register or deregister for any party by calling the people party canister. During the registration, the client specifies the latitude and longitude coordinates of the location he will be during the party.

A few minutes before a party commences, the client calls people party canister to join the party. Throughout the party, the client repeatedly calls the people party canister to obtain the state of the call. During the call, each participant streams a video that they are in the promised location. Other participants verify this by simultaneously glancing at the google street view of the location, and send an approve/decline vote to the people party canister. After the call ends, the client queries the people party canister to know if the client was approved/declined by others. The client later calls the people party canister to withdraw 1 ICP it deposited to register for the party.

===== Signaling server =====
Signaling server is used to facilitate video calls amongst the participants.

== People Party Canister Interface ==
We describe the interface supported by the people party canister here.

get_parties: () -> (vec record { PartyId; PartySpecification; }) query;
register: (id: PartyId, place: Location) -> (RegistrationResponse);
deregister: (id: PartyId) -> (DeregistrationResponse);
withdraw: (destination: AccountIdentifier) -> (WithdrawResult);
join: (id: PartyId, key: blob) -> (JoinResponse);
vote: (id: PartyId, vote: record { Vote; ParticipantName; }) -> (VoteResponse);
http_request: (request: HttpRequest) -> (HttpResponse) query;
get_person_profile: () -> (opt PersonProfile) query;
get_call_state: (id: PartyId) -> (PublicCallState) query;
get_call_result: (id: PartyId) -> (CallResult) query;

When a user enters the people party website, the frontend calls <code>get_parties</code> endpoint to display the information of all the parties. The <code>PartySpecification</code> for each people party is described as follows.
type PartySpecification =
record {
registration_start: Time;
registration_end: Time;
call_start : Time;
longitude_min : float64;
longitude_max : float64;
};

The user can then then register or deregister for a party by calling <code>register</code> and <code>deregister</code> endpoints. The canister responds with the status of registration. When the user registers for a party, he specifies the the Location (latitude and longitude) of where the user will be when the party starts.

A few minutes before the party starts, the user joins the party by calling <code>join</code> endpoint. After the user joins a call, the user sends one vote (approve/decline) for each of the other participants by calling <code>vote</code> endpoint.

At any point in time, one can call <code>get_person_profile</code>, <code>get_call_state</code> and <code>get_call_result</code> endpoints to obtain the information about the list of parties, a person's profile, state of a call and result of a call respectively.

The profile of a person is described by
type PersonProfile = record {
upcoming_parties: vec record { PartyId; Location; };
validation_score: nat64;
past_parties: vec PartyId;
};

When the user registers for a party, he needs to deposit 1 ICP. After the party ends, the user can withdraw this amount by calling <code>withdraw</code> endpoint. The user can only withdraw if he hasn't registered for any upcoming party. Once the user withdraws the amount, he cannot register again until he deposits 1 ICP.

== Party Flow ==
The below flow describing a call. Say the call is scheduled to start at 10am. Then the timeline should be as follows:

* Until 9:55am, users can enroll with the ''register'' endpoint. Since the assignment to groups has not been made, calling <code>get_call_state</code> will return what is in case 0 below.
* At 9:55am, the canister closes the registration phase and calling the <code>register</code> endpoint for the current party is not allowed anymore.
* From 9:55am to 10am, users are allowed to join. For this, there is a join endpoint that accepts a key – a blob. Calling <code>get_call_state</code> will return the result described in case 1 below.
* At 10am, the canister generates the random setup of all calls. From this time onward, calling the join endpoint is not allowed anymore, and clients are supposed to set up the WebRTC connections.
* Between 10am and 10:01am, the canister returns the call setup state as described in step 1.5 below.
* At 10:01am, the actual calls start.
* From 10:01am to 10:11am, the actual party is supposed to take place and users can submit votes. Calling <code>get_call_state</code> will return the result described in case 2 below.
* At 10:11am, the canister closes all calls and makes the tally. From this point onward, calling vote is not allowed anymore.
* From 10:11am, <code>get_call_state</code> will return the result described in case 3 below.

== Call State at Various Stages ==
The people parties canister stores the state of each call, and updates the state during each phase of the party. The canister exposes the <code>get_call_state</code> method that returns the state of the call.

==== Before Registration ====
This is returned for parties that do not exist:
call_state = record {
public_call_state = variant{not_created};
my_votes = vec{};
};

==== Joining – Before call starts ====
This is returned from the point in time where the groups are assigned but before the point in time when the call is scheduled to start. The state transition from 1 to 1.5 is defined by the canister computing the random assignment.
call_state = record {
public_call_state = variant{not_started = record {
joined = true; ← so the UI can show that the user is in the waiting room
starts_in_seconds = 30;
}};
my_votes = vec{};
};

==== Starting the call ====
call_state = record {
public_call_state = variant{starting = record{
myself = “adhesive bread”;
participants = vec{
record{
name = “adhesive bread”;
location = …
key = [30, 57, 30, …];
};
record{
Name = “stinky tofu”;
Location = …
key = [30, 57, 30, …];
...
};
starts_in_seconds = 30;
}};
my_votes = vec{};
};

== Personhood Score ==
Each identity in the Internet Computer is assigned a personhood score. If an identity hasn’t participated in any people party, its personhood score is 0. As the identity participates in people parties and is successfully validated to be a person, its personhood score increases.

Any canister will be able to query and obtain the personhood score of an identity. One can potentially use the personhood score to determine the voting power of an identity in an election/voting program.

===== Requirement #1 =====
An Internet Computer user is naturally inclined to increase his voting power. He is therefore motivated to create many identities and attend a people party with each identity. This gives him control of many identities with a positive personhood score, and thereby a higher voting power. We would like to avoid this. We would like to design the personhood score in such a way that an IC user is motivated to attend each people party with the same identity rather than creating a new identity for each people party.

===== Requirement #2 =====
Suppose Alice attended many parties with her identity, and Bob attended only one recent party. Alice and Bob’s personhood scores should still be close. Alice should not have a significant advantage by attending many parties.

===== Failed Attempt #1 =====
Suppose personhood score of an identity = 1 if the identity is successfully validated in at least one party. This personhood score violates our Requirement #1.

===== Failed Attempt #2 =====
Suppose personhood score of an identity = number of parties in which the identity is successfully validated. This personhood score violates our Requirement #2.

===== Successful Attempt =====
After a party ends, the personhood score of an identity is composed of three components:
* Recent party result (RPR): This value is 1 if the identity is accepted to be a person in the recent party. The value is 0 if the identity did not participate or is rejected in the recent party.
* Decay function of previous personhood score: The decay function is a monotonically non-increasing function. It is computed on the old personhood score (the score before the recent party started). The decay lets us give more weightage to the validation in the recent party over the previous parties.
* Bonus reward function: If the identity is accepted in the most recent party and is successfully accepted in some parties before, a bonus reward is given to the identity. This bonus component is added to incentivize IC users to participate in all the parties using the same identity rather than creating a different identity for each party.

Suppose the personhood score of an identity is PSold before a party starts. The trust score PSnew of the identity after the party ends is calculated as follows.

PSnew = RPR * (1 + bonus(PSold)) + decay(PSold)

Where Recent Party Result (RPR) is 1 iff the identity is accepted in the party.

Through empirical evidence, we found that
* bonus(x) = log10(1 + 0.25 * x) and
* decay(x) = log10(1 + x)
work pretty well. That is,

PSnew = RPR * (1 + log10(1 + 0.25 * PSold)) + log10(1 + PSold)

===== Example =====
Consider 2 scenarios.
* Scenario A: The user enters each party with the same identity, and the user is accepted in each of the parties.
* Scenario B: The user enters each party with a new identity, and the user is accepted in each of the parties.

Let us compare the personhood scores of the user in each of the scenarios. In Scenario B, we sum up the personhood scores of all the identities controlled by the user.

{| class="wikitable"
|+ Comparison of personhood scores of a user in Scenarios A & B
|-
! !! Scenario A !! Scenario B
|-
| Initial Score || 0 || 0
|-
| After 1st party || 1 || 1
|-
| After 2nd party || 1.398 || 1.301
|-
| After 3rd party || 1.510 || 1.415
|-
| After 4th party || 1.539 || 1.462
|-
| After 5th party || 1.546 || 1.482
|-
| After 6th party || 1.548 || 1.491
|-
| After 7th party || 1.548 || 1.494
|-
| After 8th party || 1.548 || 1.496
|}

Proof of Personhood

2022-12-13T17:57:30Z

Satya.vusirikala: /* Call State at Various Stages */

= This page is still work in progress =

== Proof of Personhood ==
Decentralized protocols generally rely on voting to make decisions. Anyone can create an identity and cast their vote on a proposal. A naïve implementation of this leads to a sybil attack, where one attacker creates many identities and casts many votes in his favor.

Proof of work and proof of stake are the most commonly implemented sybil attack resistance mechanisms. In proof of work, each identity has to invest some computational power to cast votes. The identity's voting power is proportional to its computational power. In proof of stake, each identity has to purchase some tokens and stake it in the protocol. The identity's voting power is proportional to its stake. In both cases, people with more money gets higher voting power and can alter the decisions of the decentralized protocol.

Proof of personhood is an alternate solution to solve the sybil attack problem. The goal of the system is to make sure each real person gets voting power of (close to) only 1. Even if the person creates many identities, he cannot have a higher voting power. In this system, anyone create an identity but its voting power is 0 by default. After the identity proves that it is controlled by a real person, it is assigned a non-zero voting power. There are many well-known techniques such as CAPTCHA to verify that someone is a real person. However, a motivated person could still manually create many identities, manually prove each of them is controlled by a real person and get a high voting power. To protect against this attack, the system needs to satisfy a second requirement -- If a single person controls many identities, the sum of voting power of all the identities is still (close to) 1. Essentially, an attacker doesn't gain any advantage by creating many identities and proving each of them is controlled by a real person. The second requirement is quite difficult to satisfy.

This article describes the ''people parties'' project currently under development at Dfinity Foundation.

== People Parties ==
People parties<ref>https://medium.com/dfinity/ultimate-decentralization-using-virtual-people-parties-that-deliver-proof-of-personhood-at-de575522c80</ref> are a mechanism for obtaining proof of personhood (also referred to as proof of humanity) built on and for the Internet Computer.

Each people party takes place at one specific point in time. Prior to that time, each prospective participant commits to a location that they will visit for the time of the party. At the beginning of the party, participants are assigned to small, random subgroups, and meet in a real-time video call with other participants assigned to the same group. The video call, however, does not show the participants’ faces; instead, participants show their surroundings, proving that they are at the place they committed to. As locations chosen by different participants must have a certain minimum distance, and no participant can be physically present at two locations simultaneously, the proof of personhood even guarantees the uniqueness of the validated persons.

The main purpose of people parties is democratization. Each validated participant can designate a neuron in the [[Network Nervous System]] that receives increased voting power; this improves the relative voting power of the “many” vs. the “heavy”. It also provides additional voting rewards to all validated participants, which further motivates people to participate in the parties. The validated personhood also benefits the Internet Computer ecosystem more broadly: Open Internet Systems, which are dapps that are controlled by a decentralized governance system, will be able to profit from the improved decentralization similarly to the Internet Computer itself. And any dapp will be able to use the validation information in order to, e.g., differentiate between bots and actual human users.

== Architecture ==
The architecture of the people parties project involves 3 components.
* Client
* People party canister
* Signaling server

===== Client =====
The client is a device that is used by the participants who want to validate their identity. The client can be a desktop, laptop or a mobile device with a web browser.

===== People party canister =====
People party canister is a canister (software program) hosted on the Internet Computer. The canister maintains the information related to all the people parties and identities. This includes
* The longitude boundaries of the participants allowed for the party
* List of participants in the party
* The state of each call in the party
* Validation results of each participant
* Personhood score (voting power) of each identity

The client queries people party canister to obtain the list of all the upcoming parties. The client can then register or deregister for any party by calling the people party canister. During the registration, the client specifies the latitude and longitude coordinates of the location he will be during the party.

A few minutes before a party commences, the client calls people party canister to join the party. Throughout the party, the client repeatedly calls the people party canister to obtain the state of the call. During the call, each participant streams a video that they are in the promised location. Other participants verify this by simultaneously glancing at the google street view of the location, and send an approve/decline vote to the people party canister. After the call ends, the client queries the people party canister to know if the client was approved/declined by others. The client later calls the people party canister to withdraw 1 ICP it deposited to register for the party.

===== Signaling server =====
Signaling server is used to facilitate video calls amongst the participants.

== People Party Canister Interface ==
We describe the interface supported by the people party canister here.

get_parties: () -> (vec record { PartyId; PartySpecification; }) query;
register: (id: PartyId, place: Location) -> (RegistrationResponse);
deregister: (id: PartyId) -> (DeregistrationResponse);
withdraw: (destination: AccountIdentifier) -> (WithdrawResult);
join: (id: PartyId, key: blob) -> (JoinResponse);
vote: (id: PartyId, vote: record { Vote; ParticipantName; }) -> (VoteResponse);
http_request: (request: HttpRequest) -> (HttpResponse) query;
get_person_profile: () -> (opt PersonProfile) query;
get_call_state: (id: PartyId) -> (PublicCallState) query;
get_call_result: (id: PartyId) -> (CallResult) query;

When a user enters the people party website, the frontend calls <code>get_parties</code> endpoint to display the information of all the parties. The <code>PartySpecification</code> for each people party is described as follows.
type PartySpecification =
record {
registration_start: Time;
registration_end: Time;
call_start : Time;
longitude_min : float64;
longitude_max : float64;
};

The user can then then register or deregister for a party by calling <code>register</code> and <code>deregister</code> endpoints. The canister responds with the status of registration. When the user registers for a party, he specifies the the Location (latitude and longitude) of where the user will be when the party starts.

A few minutes before the party starts, the user joins the party by calling <code>join</code> endpoint. After the user joins a call, the user sends one vote (approve/decline) for each of the other participants by calling <code>vote</code> endpoint.

At any point in time, one can call <code>get_person_profile</code>, <code>get_call_state</code> and <code>get_call_result</code> endpoints to obtain the information about the list of parties, a person's profile, state of a call and result of a call respectively.

The profile of a person is described by
type PersonProfile = record {
upcoming_parties: vec record { PartyId; Location; };
validation_score: nat64;
past_parties: vec PartyId;
};

When the user registers for a party, he needs to deposit 1 ICP. After the party ends, the user can withdraw this amount by calling <code>withdraw</code> endpoint. The user can only withdraw if he hasn't registered for any upcoming party. Once the user withdraws the amount, he cannot register again until he deposits 1 ICP.

== Party Flow ==
The below flow describing a call. Say the call is scheduled to start at 10am. Then the timeline should be as follows:

* Until 9:55am, users can enroll with the ''register'' endpoint. Since the assignment to groups has not been made, calling <code>get_call_state</code> will return what is in case 0 below.
* At 9:55am, the canister closes the registration phase and calling the <code>register</code> endpoint for the current party is not allowed anymore.
* From 9:55am to 10am, users are allowed to join. For this, there is a join endpoint that accepts a key – a blob. Calling <code>get_call_state</code> will return the result described in case 1 below.
* At 10am, the canister generates the random setup of all calls. From this time onward, calling the join endpoint is not allowed anymore, and clients are supposed to set up the WebRTC connections.
* Between 10am and 10:01am, the canister returns the call setup state as described in step 1.5 below.
* At 10:01am, the actual calls start.
* From 10:01am to 10:11am, the actual party is supposed to take place and users can submit votes. Calling <code>get_call_state</code> will return the result described in case 2 below.
* At 10:11am, the canister closes all calls and makes the tally. From this point onward, calling vote is not allowed anymore.
* From 10:11am, <code>get_call_state</code> will return the result described in case 3 below.

== Call State at Various Stages ==
The people parties canister stores the state of each call, and updates the state during each phase of the party. The canister exposes the <code>get_call_state</code> method that returns the state of the call.

==== Before Registration ====
This is returned for parties that do not exist:
call_state = record {
public_call_state = variant{not_created};
my_votes = vec{};
};

==== Joining – Before call starts ====
This is returned from the point in time where the groups are assigned but before the point in time when the call is scheduled to start. The state transition from 1 to 1.5 is defined by the canister computing the random assignment.
call_state = record {
public_call_state = variant{not_started = record {
joined = true; ← so the UI can show that the user is in the waiting room
starts_in_seconds = 30;
}};
my_votes = vec{};
};

== Personhood Score ==
Each identity in the Internet Computer is assigned a personhood score. If an identity hasn’t participated in any people party, its personhood score is 0. As the identity participates in people parties and is successfully validated to be a person, its personhood score increases.

Any canister will be able to query and obtain the personhood score of an identity. One can potentially use the personhood score to determine the voting power of an identity in an election/voting program.

===== Requirement #1 =====
An Internet Computer user is naturally inclined to increase his voting power. He is therefore motivated to create many identities and attend a people party with each identity. This gives him control of many identities with a positive personhood score, and thereby a higher voting power. We would like to avoid this. We would like to design the personhood score in such a way that an IC user is motivated to attend each people party with the same identity rather than creating a new identity for each people party.

===== Requirement #2 =====
Suppose Alice attended many parties with her identity, and Bob attended only one recent party. Alice and Bob’s personhood scores should still be close. Alice should not have a significant advantage by attending many parties.

===== Failed Attempt #1 =====
Suppose personhood score of an identity = 1 if the identity is successfully validated in at least one party. This personhood score violates our Requirement #1.

===== Failed Attempt #2 =====
Suppose personhood score of an identity = number of parties in which the identity is successfully validated. This personhood score violates our Requirement #2.

===== Successful Attempt =====
After a party ends, the personhood score of an identity is composed of three components:
* Recent party result (RPR): This value is 1 if the identity is accepted to be a person in the recent party. The value is 0 if the identity did not participate or is rejected in the recent party.
* Decay function of previous personhood score: The decay function is a monotonically non-increasing function. It is computed on the old personhood score (the score before the recent party started). The decay lets us give more weightage to the validation in the recent party over the previous parties.
* Bonus reward function: If the identity is accepted in the most recent party and is successfully accepted in some parties before, a bonus reward is given to the identity. This bonus component is added to incentivize IC users to participate in all the parties using the same identity rather than creating a different identity for each party.

Suppose the personhood score of an identity is PSold before a party starts. The trust score PSnew of the identity after the party ends is calculated as follows.

PSnew = RPR * (1 + bonus(PSold)) + decay(PSold)

Where Recent Party Result (RPR) is 1 iff the identity is accepted in the party.

Through empirical evidence, we found that
* bonus(x) = log10(1 + 0.25 * x) and
* decay(x) = log10(1 + x)
work pretty well. That is,

PSnew = RPR * (1 + log10(1 + 0.25 * PSold)) + log10(1 + PSold)

===== Example =====
Consider 2 scenarios.
* Scenario A: The user enters each party with the same identity, and the user is accepted in each of the parties.
* Scenario B: The user enters each party with a new identity, and the user is accepted in each of the parties.

Let us compare the personhood scores of the user in each of the scenarios. In Scenario B, we sum up the personhood scores of all the identities controlled by the user.

{| class="wikitable"
|+ Comparison of personhood scores of a user in Scenarios A & B
|-
! !! Scenario A !! Scenario B
|-
| Initial Score || 0 || 0
|-
| After 1st party || 1 || 1
|-
| After 2nd party || 1.398 || 1.301
|-
| After 3rd party || 1.510 || 1.415
|-
| After 4th party || 1.539 || 1.462
|-
| After 5th party || 1.546 || 1.482
|-
| After 6th party || 1.548 || 1.491
|-
| After 7th party || 1.548 || 1.494
|-
| After 8th party || 1.548 || 1.496
|}

Proof of Personhood

2022-12-13T17:56:45Z

Satya.vusirikala: /* Call State at Various Stages */

= This page is still work in progress =

== Proof of Personhood ==
Decentralized protocols generally rely on voting to make decisions. Anyone can create an identity and cast their vote on a proposal. A naïve implementation of this leads to a sybil attack, where one attacker creates many identities and casts many votes in his favor.

Proof of work and proof of stake are the most commonly implemented sybil attack resistance mechanisms. In proof of work, each identity has to invest some computational power to cast votes. The identity's voting power is proportional to its computational power. In proof of stake, each identity has to purchase some tokens and stake it in the protocol. The identity's voting power is proportional to its stake. In both cases, people with more money gets higher voting power and can alter the decisions of the decentralized protocol.

Proof of personhood is an alternate solution to solve the sybil attack problem. The goal of the system is to make sure each real person gets voting power of (close to) only 1. Even if the person creates many identities, he cannot have a higher voting power. In this system, anyone create an identity but its voting power is 0 by default. After the identity proves that it is controlled by a real person, it is assigned a non-zero voting power. There are many well-known techniques such as CAPTCHA to verify that someone is a real person. However, a motivated person could still manually create many identities, manually prove each of them is controlled by a real person and get a high voting power. To protect against this attack, the system needs to satisfy a second requirement -- If a single person controls many identities, the sum of voting power of all the identities is still (close to) 1. Essentially, an attacker doesn't gain any advantage by creating many identities and proving each of them is controlled by a real person. The second requirement is quite difficult to satisfy.

This article describes the ''people parties'' project currently under development at Dfinity Foundation.

== People Parties ==
People parties<ref>https://medium.com/dfinity/ultimate-decentralization-using-virtual-people-parties-that-deliver-proof-of-personhood-at-de575522c80</ref> are a mechanism for obtaining proof of personhood (also referred to as proof of humanity) built on and for the Internet Computer.

Each people party takes place at one specific point in time. Prior to that time, each prospective participant commits to a location that they will visit for the time of the party. At the beginning of the party, participants are assigned to small, random subgroups, and meet in a real-time video call with other participants assigned to the same group. The video call, however, does not show the participants’ faces; instead, participants show their surroundings, proving that they are at the place they committed to. As locations chosen by different participants must have a certain minimum distance, and no participant can be physically present at two locations simultaneously, the proof of personhood even guarantees the uniqueness of the validated persons.

The main purpose of people parties is democratization. Each validated participant can designate a neuron in the [[Network Nervous System]] that receives increased voting power; this improves the relative voting power of the “many” vs. the “heavy”. It also provides additional voting rewards to all validated participants, which further motivates people to participate in the parties. The validated personhood also benefits the Internet Computer ecosystem more broadly: Open Internet Systems, which are dapps that are controlled by a decentralized governance system, will be able to profit from the improved decentralization similarly to the Internet Computer itself. And any dapp will be able to use the validation information in order to, e.g., differentiate between bots and actual human users.

== Architecture ==
The architecture of the people parties project involves 3 components.
* Client
* People party canister
* Signaling server

===== Client =====
The client is a device that is used by the participants who want to validate their identity. The client can be a desktop, laptop or a mobile device with a web browser.

===== People party canister =====
People party canister is a canister (software program) hosted on the Internet Computer. The canister maintains the information related to all the people parties and identities. This includes
* The longitude boundaries of the participants allowed for the party
* List of participants in the party
* The state of each call in the party
* Validation results of each participant
* Personhood score (voting power) of each identity

The client queries people party canister to obtain the list of all the upcoming parties. The client can then register or deregister for any party by calling the people party canister. During the registration, the client specifies the latitude and longitude coordinates of the location he will be during the party.

A few minutes before a party commences, the client calls people party canister to join the party. Throughout the party, the client repeatedly calls the people party canister to obtain the state of the call. During the call, each participant streams a video that they are in the promised location. Other participants verify this by simultaneously glancing at the google street view of the location, and send an approve/decline vote to the people party canister. After the call ends, the client queries the people party canister to know if the client was approved/declined by others. The client later calls the people party canister to withdraw 1 ICP it deposited to register for the party.

===== Signaling server =====
Signaling server is used to facilitate video calls amongst the participants.

== People Party Canister Interface ==
We describe the interface supported by the people party canister here.

get_parties: () -> (vec record { PartyId; PartySpecification; }) query;
register: (id: PartyId, place: Location) -> (RegistrationResponse);
deregister: (id: PartyId) -> (DeregistrationResponse);
withdraw: (destination: AccountIdentifier) -> (WithdrawResult);
join: (id: PartyId, key: blob) -> (JoinResponse);
vote: (id: PartyId, vote: record { Vote; ParticipantName; }) -> (VoteResponse);
http_request: (request: HttpRequest) -> (HttpResponse) query;
get_person_profile: () -> (opt PersonProfile) query;
get_call_state: (id: PartyId) -> (PublicCallState) query;
get_call_result: (id: PartyId) -> (CallResult) query;

When a user enters the people party website, the frontend calls <code>get_parties</code> endpoint to display the information of all the parties. The <code>PartySpecification</code> for each people party is described as follows.
type PartySpecification =
record {
registration_start: Time;
registration_end: Time;
call_start : Time;
longitude_min : float64;
longitude_max : float64;
};

The user can then then register or deregister for a party by calling <code>register</code> and <code>deregister</code> endpoints. The canister responds with the status of registration. When the user registers for a party, he specifies the the Location (latitude and longitude) of where the user will be when the party starts.

A few minutes before the party starts, the user joins the party by calling <code>join</code> endpoint. After the user joins a call, the user sends one vote (approve/decline) for each of the other participants by calling <code>vote</code> endpoint.

At any point in time, one can call <code>get_person_profile</code>, <code>get_call_state</code> and <code>get_call_result</code> endpoints to obtain the information about the list of parties, a person's profile, state of a call and result of a call respectively.

The profile of a person is described by
type PersonProfile = record {
upcoming_parties: vec record { PartyId; Location; };
validation_score: nat64;
past_parties: vec PartyId;
};

When the user registers for a party, he needs to deposit 1 ICP. After the party ends, the user can withdraw this amount by calling <code>withdraw</code> endpoint. The user can only withdraw if he hasn't registered for any upcoming party. Once the user withdraws the amount, he cannot register again until he deposits 1 ICP.

== Party Flow ==
The below flow describing a call. Say the call is scheduled to start at 10am. Then the timeline should be as follows:

* Until 9:55am, users can enroll with the ''register'' endpoint. Since the assignment to groups has not been made, calling <code>get_call_state</code> will return what is in case 0 below.
* At 9:55am, the canister closes the registration phase and calling the <code>register</code> endpoint for the current party is not allowed anymore.
* From 9:55am to 10am, users are allowed to join. For this, there is a join endpoint that accepts a key – a blob. Calling <code>get_call_state</code> will return the result described in case 1 below.
* At 10am, the canister generates the random setup of all calls. From this time onward, calling the join endpoint is not allowed anymore, and clients are supposed to set up the WebRTC connections.
* Between 10am and 10:01am, the canister returns the call setup state as described in step 1.5 below.
* At 10:01am, the actual calls start.
* From 10:01am to 10:11am, the actual party is supposed to take place and users can submit votes. Calling <code>get_call_state</code> will return the result described in case 2 below.
* At 10:11am, the canister closes all calls and makes the tally. From this point onward, calling vote is not allowed anymore.
* From 10:11am, <code>get_call_state</code> will return the result described in case 3 below.

== Call State at Various Stages ==
The people parties canister stores the state of each call, and updates the state during each phase of the party. The canister exposes the <code>get_call_state</code> method that returns the state of the call.

==== Before Registration ====
This is returned for parties that do not exist:
call_state = record {
public_call_state = variant{not_created};
my_votes = vec{};
};

== Personhood Score ==
Each identity in the Internet Computer is assigned a personhood score. If an identity hasn’t participated in any people party, its personhood score is 0. As the identity participates in people parties and is successfully validated to be a person, its personhood score increases.

Any canister will be able to query and obtain the personhood score of an identity. One can potentially use the personhood score to determine the voting power of an identity in an election/voting program.

===== Requirement #1 =====
An Internet Computer user is naturally inclined to increase his voting power. He is therefore motivated to create many identities and attend a people party with each identity. This gives him control of many identities with a positive personhood score, and thereby a higher voting power. We would like to avoid this. We would like to design the personhood score in such a way that an IC user is motivated to attend each people party with the same identity rather than creating a new identity for each people party.

===== Requirement #2 =====
Suppose Alice attended many parties with her identity, and Bob attended only one recent party. Alice and Bob’s personhood scores should still be close. Alice should not have a significant advantage by attending many parties.

===== Failed Attempt #1 =====
Suppose personhood score of an identity = 1 if the identity is successfully validated in at least one party. This personhood score violates our Requirement #1.

===== Failed Attempt #2 =====
Suppose personhood score of an identity = number of parties in which the identity is successfully validated. This personhood score violates our Requirement #2.

===== Successful Attempt =====
After a party ends, the personhood score of an identity is composed of three components:
* Recent party result (RPR): This value is 1 if the identity is accepted to be a person in the recent party. The value is 0 if the identity did not participate or is rejected in the recent party.
* Decay function of previous personhood score: The decay function is a monotonically non-increasing function. It is computed on the old personhood score (the score before the recent party started). The decay lets us give more weightage to the validation in the recent party over the previous parties.
* Bonus reward function: If the identity is accepted in the most recent party and is successfully accepted in some parties before, a bonus reward is given to the identity. This bonus component is added to incentivize IC users to participate in all the parties using the same identity rather than creating a different identity for each party.

Suppose the personhood score of an identity is PSold before a party starts. The trust score PSnew of the identity after the party ends is calculated as follows.

PSnew = RPR * (1 + bonus(PSold)) + decay(PSold)

Where Recent Party Result (RPR) is 1 iff the identity is accepted in the party.

Through empirical evidence, we found that
* bonus(x) = log10(1 + 0.25 * x) and
* decay(x) = log10(1 + x)
work pretty well. That is,

PSnew = RPR * (1 + log10(1 + 0.25 * PSold)) + log10(1 + PSold)

===== Example =====
Consider 2 scenarios.
* Scenario A: The user enters each party with the same identity, and the user is accepted in each of the parties.
* Scenario B: The user enters each party with a new identity, and the user is accepted in each of the parties.

Let us compare the personhood scores of the user in each of the scenarios. In Scenario B, we sum up the personhood scores of all the identities controlled by the user.

{| class="wikitable"
|+ Comparison of personhood scores of a user in Scenarios A & B
|-
! !! Scenario A !! Scenario B
|-
| Initial Score || 0 || 0
|-
| After 1st party || 1 || 1
|-
| After 2nd party || 1.398 || 1.301
|-
| After 3rd party || 1.510 || 1.415
|-
| After 4th party || 1.539 || 1.462
|-
| After 5th party || 1.546 || 1.482
|-
| After 6th party || 1.548 || 1.491
|-
| After 7th party || 1.548 || 1.494
|-
| After 8th party || 1.548 || 1.496
|}

Proof of Personhood

2022-12-13T17:56:15Z

Satya.vusirikala:

= This page is still work in progress =

== Proof of Personhood ==
Decentralized protocols generally rely on voting to make decisions. Anyone can create an identity and cast their vote on a proposal. A naïve implementation of this leads to a sybil attack, where one attacker creates many identities and casts many votes in his favor.

Proof of work and proof of stake are the most commonly implemented sybil attack resistance mechanisms. In proof of work, each identity has to invest some computational power to cast votes. The identity's voting power is proportional to its computational power. In proof of stake, each identity has to purchase some tokens and stake it in the protocol. The identity's voting power is proportional to its stake. In both cases, people with more money gets higher voting power and can alter the decisions of the decentralized protocol.

Proof of personhood is an alternate solution to solve the sybil attack problem. The goal of the system is to make sure each real person gets voting power of (close to) only 1. Even if the person creates many identities, he cannot have a higher voting power. In this system, anyone create an identity but its voting power is 0 by default. After the identity proves that it is controlled by a real person, it is assigned a non-zero voting power. There are many well-known techniques such as CAPTCHA to verify that someone is a real person. However, a motivated person could still manually create many identities, manually prove each of them is controlled by a real person and get a high voting power. To protect against this attack, the system needs to satisfy a second requirement -- If a single person controls many identities, the sum of voting power of all the identities is still (close to) 1. Essentially, an attacker doesn't gain any advantage by creating many identities and proving each of them is controlled by a real person. The second requirement is quite difficult to satisfy.

This article describes the ''people parties'' project currently under development at Dfinity Foundation.

== People Parties ==
People parties<ref>https://medium.com/dfinity/ultimate-decentralization-using-virtual-people-parties-that-deliver-proof-of-personhood-at-de575522c80</ref> are a mechanism for obtaining proof of personhood (also referred to as proof of humanity) built on and for the Internet Computer.

Each people party takes place at one specific point in time. Prior to that time, each prospective participant commits to a location that they will visit for the time of the party. At the beginning of the party, participants are assigned to small, random subgroups, and meet in a real-time video call with other participants assigned to the same group. The video call, however, does not show the participants’ faces; instead, participants show their surroundings, proving that they are at the place they committed to. As locations chosen by different participants must have a certain minimum distance, and no participant can be physically present at two locations simultaneously, the proof of personhood even guarantees the uniqueness of the validated persons.

The main purpose of people parties is democratization. Each validated participant can designate a neuron in the [[Network Nervous System]] that receives increased voting power; this improves the relative voting power of the “many” vs. the “heavy”. It also provides additional voting rewards to all validated participants, which further motivates people to participate in the parties. The validated personhood also benefits the Internet Computer ecosystem more broadly: Open Internet Systems, which are dapps that are controlled by a decentralized governance system, will be able to profit from the improved decentralization similarly to the Internet Computer itself. And any dapp will be able to use the validation information in order to, e.g., differentiate between bots and actual human users.

== Architecture ==
The architecture of the people parties project involves 3 components.
* Client
* People party canister
* Signaling server

===== Client =====
The client is a device that is used by the participants who want to validate their identity. The client can be a desktop, laptop or a mobile device with a web browser.

===== People party canister =====
People party canister is a canister (software program) hosted on the Internet Computer. The canister maintains the information related to all the people parties and identities. This includes
* The longitude boundaries of the participants allowed for the party
* List of participants in the party
* The state of each call in the party
* Validation results of each participant
* Personhood score (voting power) of each identity

The client queries people party canister to obtain the list of all the upcoming parties. The client can then register or deregister for any party by calling the people party canister. During the registration, the client specifies the latitude and longitude coordinates of the location he will be during the party.

A few minutes before a party commences, the client calls people party canister to join the party. Throughout the party, the client repeatedly calls the people party canister to obtain the state of the call. During the call, each participant streams a video that they are in the promised location. Other participants verify this by simultaneously glancing at the google street view of the location, and send an approve/decline vote to the people party canister. After the call ends, the client queries the people party canister to know if the client was approved/declined by others. The client later calls the people party canister to withdraw 1 ICP it deposited to register for the party.

===== Signaling server =====
Signaling server is used to facilitate video calls amongst the participants.

== People Party Canister Interface ==
We describe the interface supported by the people party canister here.

get_parties: () -> (vec record { PartyId; PartySpecification; }) query;
register: (id: PartyId, place: Location) -> (RegistrationResponse);
deregister: (id: PartyId) -> (DeregistrationResponse);
withdraw: (destination: AccountIdentifier) -> (WithdrawResult);
join: (id: PartyId, key: blob) -> (JoinResponse);
vote: (id: PartyId, vote: record { Vote; ParticipantName; }) -> (VoteResponse);
http_request: (request: HttpRequest) -> (HttpResponse) query;
get_person_profile: () -> (opt PersonProfile) query;
get_call_state: (id: PartyId) -> (PublicCallState) query;
get_call_result: (id: PartyId) -> (CallResult) query;

When a user enters the people party website, the frontend calls <code>get_parties</code> endpoint to display the information of all the parties. The <code>PartySpecification</code> for each people party is described as follows.
type PartySpecification =
record {
registration_start: Time;
registration_end: Time;
call_start : Time;
longitude_min : float64;
longitude_max : float64;
};

The user can then then register or deregister for a party by calling <code>register</code> and <code>deregister</code> endpoints. The canister responds with the status of registration. When the user registers for a party, he specifies the the Location (latitude and longitude) of where the user will be when the party starts.

A few minutes before the party starts, the user joins the party by calling <code>join</code> endpoint. After the user joins a call, the user sends one vote (approve/decline) for each of the other participants by calling <code>vote</code> endpoint.

At any point in time, one can call <code>get_person_profile</code>, <code>get_call_state</code> and <code>get_call_result</code> endpoints to obtain the information about the list of parties, a person's profile, state of a call and result of a call respectively.

The profile of a person is described by
type PersonProfile = record {
upcoming_parties: vec record { PartyId; Location; };
validation_score: nat64;
past_parties: vec PartyId;
};

When the user registers for a party, he needs to deposit 1 ICP. After the party ends, the user can withdraw this amount by calling <code>withdraw</code> endpoint. The user can only withdraw if he hasn't registered for any upcoming party. Once the user withdraws the amount, he cannot register again until he deposits 1 ICP.

== Party Flow ==
The below flow describing a call. Say the call is scheduled to start at 10am. Then the timeline should be as follows:

* Until 9:55am, users can enroll with the ''register'' endpoint. Since the assignment to groups has not been made, calling <code>get_call_state</code> will return what is in case 0 below.
* At 9:55am, the canister closes the registration phase and calling the <code>register</code> endpoint for the current party is not allowed anymore.
* From 9:55am to 10am, users are allowed to join. For this, there is a join endpoint that accepts a key – a blob. Calling <code>get_call_state</code> will return the result described in case 1 below.
* At 10am, the canister generates the random setup of all calls. From this time onward, calling the join endpoint is not allowed anymore, and clients are supposed to set up the WebRTC connections.
* Between 10am and 10:01am, the canister returns the call setup state as described in step 1.5 below.
* At 10:01am, the actual calls start.
* From 10:01am to 10:11am, the actual party is supposed to take place and users can submit votes. Calling <code>get_call_state</code> will return the result described in case 2 below.
* At 10:11am, the canister closes all calls and makes the tally. From this point onward, calling vote is not allowed anymore.
* From 10:11am, <code>get_call_state</code> will return the result described in case 3 below.

== Call State at Various Stages ==
The people parties canister stores the state of each call, and updates the state during each phase of the party. The canister exposes the <code>get_call_state</code> method that returns the state of the call.

=== Before Registration ===
This is returned for parties that do not exist:
call_state = record {
public_call_state = variant{not_created};
my_votes = vec{};
};

== Personhood Score ==
Each identity in the Internet Computer is assigned a personhood score. If an identity hasn’t participated in any people party, its personhood score is 0. As the identity participates in people parties and is successfully validated to be a person, its personhood score increases.

Any canister will be able to query and obtain the personhood score of an identity. One can potentially use the personhood score to determine the voting power of an identity in an election/voting program.

===== Requirement #1 =====
An Internet Computer user is naturally inclined to increase his voting power. He is therefore motivated to create many identities and attend a people party with each identity. This gives him control of many identities with a positive personhood score, and thereby a higher voting power. We would like to avoid this. We would like to design the personhood score in such a way that an IC user is motivated to attend each people party with the same identity rather than creating a new identity for each people party.

===== Requirement #2 =====
Suppose Alice attended many parties with her identity, and Bob attended only one recent party. Alice and Bob’s personhood scores should still be close. Alice should not have a significant advantage by attending many parties.

===== Failed Attempt #1 =====
Suppose personhood score of an identity = 1 if the identity is successfully validated in at least one party. This personhood score violates our Requirement #1.

===== Failed Attempt #2 =====
Suppose personhood score of an identity = number of parties in which the identity is successfully validated. This personhood score violates our Requirement #2.

===== Successful Attempt =====
After a party ends, the personhood score of an identity is composed of three components:
* Recent party result (RPR): This value is 1 if the identity is accepted to be a person in the recent party. The value is 0 if the identity did not participate or is rejected in the recent party.
* Decay function of previous personhood score: The decay function is a monotonically non-increasing function. It is computed on the old personhood score (the score before the recent party started). The decay lets us give more weightage to the validation in the recent party over the previous parties.
* Bonus reward function: If the identity is accepted in the most recent party and is successfully accepted in some parties before, a bonus reward is given to the identity. This bonus component is added to incentivize IC users to participate in all the parties using the same identity rather than creating a different identity for each party.

Suppose the personhood score of an identity is PSold before a party starts. The trust score PSnew of the identity after the party ends is calculated as follows.

PSnew = RPR * (1 + bonus(PSold)) + decay(PSold)

Where Recent Party Result (RPR) is 1 iff the identity is accepted in the party.

Through empirical evidence, we found that
* bonus(x) = log10(1 + 0.25 * x) and
* decay(x) = log10(1 + x)
work pretty well. That is,

PSnew = RPR * (1 + log10(1 + 0.25 * PSold)) + log10(1 + PSold)

===== Example =====
Consider 2 scenarios.
* Scenario A: The user enters each party with the same identity, and the user is accepted in each of the parties.
* Scenario B: The user enters each party with a new identity, and the user is accepted in each of the parties.

Let us compare the personhood scores of the user in each of the scenarios. In Scenario B, we sum up the personhood scores of all the identities controlled by the user.

{| class="wikitable"
|+ Comparison of personhood scores of a user in Scenarios A & B
|-
! !! Scenario A !! Scenario B
|-
| Initial Score || 0 || 0
|-
| After 1st party || 1 || 1
|-
| After 2nd party || 1.398 || 1.301
|-
| After 3rd party || 1.510 || 1.415
|-
| After 4th party || 1.539 || 1.462
|-
| After 5th party || 1.546 || 1.482
|-
| After 6th party || 1.548 || 1.491
|-
| After 7th party || 1.548 || 1.494
|-
| After 8th party || 1.548 || 1.496
|}

New Subnet Creation

2022-11-28T23:58:16Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to subnet list.
*** For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** Store subnet record.
*** Add a key-value pair with key = <code>subnet_record_{subnet_id}</code> and value = newly created subnet record, where {subnet_id} is the subnet id of the newly created subnet.
** Store catch-up package.
*** Add a key-value pair with key = <code>catch_up_package_contents_{subnet_id}</code> and value = newly created catch-up package.
** Store subnet threshold signing key.
*** Add a key-value pair with key = <code>crypto_threshold_signing_public_key_{subnet_id}</code> and value = threshold BLS signing key generated above.
** Routing table mutation.
*** Each subnet has a range of canister ids that are utilized for canisters in that subnet. The range of canister ids for each subnet is called the routing table. The routing table helps us route canister traffic to the appropriate subnet. When a new subnet is created, the next available set of <code>CANISTER_IDS_PER_SUBNET</code> (currently set to 2^20) canister ids is assigned as the range for the subnet. This information is stored in the key <code>routing_table</code> of the registry.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:42:08Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to subnet list.
*** For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** Store subnet record.
*** Add a key-value pair with key = <code>subnet_record_{subnet_id}</code> and value = newly created subnet record, where {subnet_id} is the subnet id of the newly created subnet.
** Store catch-up package.
*** Add a key-value pair with key = <code>catch_up_package_contents_{subnet_id}</code> and value = newly created catch-up package.
** Store subnet threshold signing key.
*** Add a key-value pair with key = <code>crypto_threshold_signing_public_key_{subnet_id}</code> and value = threshold BLS signing key generated above.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:41:17Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to subnet list. For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** Store subnet record. Add a key-value pair with key = <code>subnet_record_{subnet_id}</code> and value = newly created subnet record, where {subnet_id} is the subnet id of the newly created subnet.
** Store catch-up package. Add a key-value pair with key = <code>catch_up_package_contents_{subnet_id}</code> and value = newly created catch-up package.
** Store subnet threshold signing key. Add a key-value pair with key = <code>crypto_threshold_signing_public_key_{subnet_id}</code> and value = threshold BLS signing key generated above.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:39:18Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to subnet list. For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** Store subnet record. Add a key-value pair with key = <code>subnet_record_{subnet_id}</code> and value = newly created subnet record, where {subnet_id} is the subnet id of the newly created subnet.
** Store catch-up package. Add a key-value pair with key = <code> catch_up_package_contents_{subnet_id}</code> and value = newly created catch-up package.
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:38:46Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to Subnet list. For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** Store subnet record. Add a key-value pair with key = <code>subnet_record_{subnet_id}</code> and value = newly created subnet record, where {subnet_id} is the subnet id of the newly created subnet.
** Store catch-up package. Add a key-value pair with key = <code> catch_up_package_contents_{subnet_id}</code> and value = newly created catch-up package.
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:37:59Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to Subnet list. For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** New subnet. Add a key-value pair with key = <code>subnet_record_{subnet_id}</code> and value = newly created subnet record, where {subnet_id} is the subnet id of the newly created subnet.
** New subnet Distributed Key Generation (DKG). Add a key-value pair with key = <code> catch_up_package_contents_{subnet_id}</code> and value = newly created catch-up package.
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:35:59Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to Subnet list. For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** New subnet. Add a key-value pair with key = <code>subnet_record_<subnet_id></code> and value = newly created subnet record, where <subnet_id> is the subnet id of the newly created subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:35:24Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to Subnet list. For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** New subnet. Add a key-value pair with key = <code>subnet_record_<subnet_id></code> and value = newly created subnet record.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:35:07Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to Subnet list. For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** New subnet. Add a key-value pair with key = <code>subnet_record_\<subnet_id\></code> and value = newly created subnet record.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:31:10Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry. The registry is a versioned key-value storage.
** Append to Subnet list. For the key <code>subnet_list</code>, the registry stores the list of all subnets. Append the new subnet to this list.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:24:17Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
** Each subnet has a subnet record storing all its configuration information. The subnet record of the new subnet contains all the information present in the payload of the 'Create Subnet' proposal.
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:20:47Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[Summary Block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:19:58Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a genesis block and [[Catch-Up Package]]
** The registry canister creates a genesis block. This is the first block in the blockchain. Structurally, this is a [[summary block]] containing the DKG dealings computed above.
** The registry canister then generates a Catch-Up Package which contains the above genesis block along with other information required for the subnet to start the consensus protocol.
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:15:26Z

Satya.vusirikala: /* Payload for 'Create Subnet' Proposal */

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block
** ecdsa_config - This field is optional. It specifies the configuration settings to be used for ECDSA signature scheme.
** dkg_interval_length - This is the number of rounds in an epoch. At the end of each epoch, the nodes run a distributed key generation protocol to reshare their secret key shares.
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a [[Catch-Up Package]]
** A Catch-Up Package for a subnet at a particular height contains the relevant information for a node to start consensus from that height.
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T23:11:30Z

Satya.vusirikala:

Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a [[Catch-Up Package]]
** A Catch-Up Package for a subnet at a particular height contains the relevant information for a node to start consensus from that height.
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T22:07:11Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a [[Catch-Up Package]]
** A Catch-Up Package for a subnet at a particular height contains the relevant information for a node to start consensus from that height.
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T22:03:20Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the [[Distributed Key Generation]] protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key. A subnet also utilizes threshold BLS signature scheme for many other purposes including the generation of random beacon in the consensus layer and generation of random tape in the execution layer.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol. At the end of the protocol, all the nodes in the NNS agree on a few "dealings" for each node in the new subnet. The dealings of a node can be combined to obtain the signing key share of the node. The dealings for each node are encrypted with the encryption key registered by the node.
* Create a [[Catch-Up Package]]
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T21:55:20Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the Distributed Key Generation protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key.
** In order to generate the above signing key share for each node in the new subnet, the nodes in the NNS engage in a distributed key generation (DKG) protocol.
* Create a [[Catch-Up Package]]
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T20:53:04Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the Distributed Key Generation protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key (none of the nodes know the complete subnet's signing key). After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature "shares" can be combined to obtain the signature of the replicated state using the subnet's signing key.
**
* Create a [[Catch-Up Package]]
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T20:49:46Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the Distributed Key Generation protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. After every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. For this, the subnet utilizes a threshold BLS signature scheme. In a nutshell, each node in the subnet holds a "share" of the subnet's signing key. After every round, each of the nodes individually sign their replicated state using their signing key "share". If at least a "threshold" number of nodes agree upon the same replicated state, their signature shares can be combined to obtain the signature of the replicated state using the subnet's signing key.
* Create a [[Catch-Up Package]]
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T20:43:21Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the Distributed Key Generation protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. In every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest [https://wiki.internetcomputer.org/wiki/Replicated_state_structure replicated state]. A subnet utilizes a threshold BLS signing key to sign
* Create a [[Catch-Up Package]]
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T20:42:29Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the Distributed Key Generation protocol.
** The first step is to generate the cryptographic information required for the new subnet to work properly. In every round of the Internet Computer protocol, the nodes of the subnet agree upon and sign their latest replicated state. A subnet utilizes a threshold BLS signing key to sign
* Create a [[Catch-Up Package]]
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-28T20:33:43Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The registry canister performs the following steps.
* Run the Distributed Key Generation protocol.
* Create a [[Catch-Up Package]]
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-25T14:47:12Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

* Perform Distributed Key Generation protocol
* Create a [[Catch-Up Package]]
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-25T14:44:27Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

* Perform Distributed Key Generation protocol
* Create a Catch-Up Package
* Create a Subnet Record
* Perform the following 5 mutations to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-25T14:43:57Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

* Perform Distributed Key Generation protocol
* Create a Catch-Up Package
* Create a Subnet Record
* The following 5 mutations are made to the registry.
** Subnet list mutation.
** New subnet.
** New subnet Distributed Key Generation (DKG).
** New subnet threshold signing key.
** Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-25T14:40:31Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage. Each mutation to the registry is stored as a new version.

The following 5 mutations are made to the registry.
* Subnet list mutation.
* New subnet.
* New subnet Distributed Key Generation (DKG).
* New subnet threshold signing key.
* Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-25T14:39:56Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage.

The following 5 mutations are made to the registry.
* Subnet list mutation.
* New subnet.
* New subnet Distributed Key Generation (DKG).
* New subnet threshold signing key.
* Routing table mutation.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-25T14:37:56Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add configuration information of the new subnet. The registry is a versioned key-value storage.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-25T14:36:12Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]), which mutates the registry to add information about the new subnet.

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-25T14:26:31Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, the proposal is executed automatically. To execute the proposal, a method in the registry canister is called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]).

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-25T14:25:06Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==
When the 'Create Subnet' NNS proposal is voted and accepted by the community, a method in the registry canister is automatically called ([https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github]).

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-23T23:58:08Z

Satya.vusirikala: /* Creating a New Subnet */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process ([https://github.com/dfinity/ic/tree/master/rs/orchestrator/registry_replicator github]) which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-23T23:53:18Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-23T23:52:48Z

Satya.vusirikala: /* Creating a New Subnet */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record.

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-23T23:52:28Z

Satya.vusirikala: /* Creating a New Subnet */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record. The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software ([[#Installing_the_Replica]]) and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-23T23:51:44Z

Satya.vusirikala: /* Installing the Replica */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record. The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. If a node is not assigned to any subnet, both the replica partitions are empty. If the node is actively participating in a subnet, one of the replica partitions is used to run the replica process and the other replica partition is empty. To install the new replica software,
* The orchestrator loads the replica image into an empty replica partition.
* The orchestrator then modifies the boot loader settings to use the newly loaded replica partition.
* The orchestrator restarts the node.
* When the node boots up again, the newly loaded replica partition is used as per the boot loader settings.

New Subnet Creation

2022-11-23T23:32:49Z

Satya.vusirikala: /* Installing the Replica */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record. The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. One of the replica partition is usually active (being used) and the other replica partition is empty. When the orchestrator

New Subnet Creation

2022-11-23T23:29:07Z

Satya.vusirikala: /* Executing the 'Create Subnet' Proposal */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record. The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

== Installing the Replica ==
The orchestrator uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software, which works as follows. The disk is divided into 3 partitions -- 2 partitions for the replica process and 1 partition for the data. One of the replica partition is usually active/being used and the other replica partition is empty. When the orchestrator

New Subnet Creation

2022-11-23T23:28:20Z

Satya.vusirikala: /* Creating a New Subnet */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record. The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now installs the replica software and restarts the node.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

New Subnet Creation

2022-11-23T23:24:46Z

Satya.vusirikala: /* Creating a New Subnet */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record. The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration. The orchestrator now uses a [https://en.wikipedia.org/wiki/Blue-green_deployment blue/green deployment] to start the replica software.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

New Subnet Creation

2022-11-23T23:20:07Z

Satya.vusirikala: /* Creating a New Subnet */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record. The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a new subnet according to the registry, the orchestrator first downloads the replica software using the url in the subnet's registry configuration.

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==

New Subnet Creation

2022-11-23T23:18:03Z

Satya.vusirikala: /* Creating a New Subnet */

= This Page is Still Work in Progress =
Ever wondered about the meaning behind DFINITY? It’s Decentralized + Infinity. It’s named that way because the Internet Computer is designed to scale infinitely. It means that the Internet Computer can host an unlimited number of canisters (smart contracts), store an unlimited amount of memory, process an unlimited amount of transactions per second. In simple words, Internet Computer is designed to host even large scale social media platforms in a fully decentralized way.

There are two types of widely-used approaches to improve the scalability of a system. (1) Vertical Scaling, and (2) Horizontal Scaling. Vertical scaling means adding more CPU, RAM and disk to a single computer. Horizontal scaling means adding more computers to the system. There is a limit to vertical scaling. But with horizontal scaling, one can achieve unlimited scalability. Internet Computer is one of the first blockchains to successfully use horizontal scaling.

The nodes in the Internet Computer are divided into subnets, each containing a few dozen nodes. The set of nodes in a subnet together maintain one blockchain. Each subnet can host a few thousand canisters and process messages received by those canisters. Each subnet has a limited capacity in terms of the number of canisters (a few thousand), amount of storage (a few TB), and bandwidth (a few hundred transactions per second). But as more subnets are added to the Internet Computer, its overall capacity increases proportionately. There is no limit on the number of subnets that can be added, resulting in unlimited scalability.

We need to do 2 things to create a new subnet.
* Add/Register new nodes.
* Create a new subnet with the registered nodes that were not yet assigned to any subnet.

== Adding/Registering New Nodes ==
We new describe a series of steps that need to be followed to add a new node to the Internet Computer.
* Node provider purchases a NitroKey (a Hardware Security Module), generates a public-key/secret-key pair, and submits an NNS proposal to add his public key to the NNS registry. The community votes on the proposal. If the majority accept the proposal, then the node provider's credentials are added to the NNS registry. From now on, the NNS canisters trust the messages signed by the node provider's secret key. The entire process is specified in the [https://wiki.internetcomputer.org/wiki/Node_Provider_Onboarding node provider onboarding article].
* Node provider purchases node hardware with the recommended specifications and places it in a data center rack that meets the recommended specifications.
* The node doesn't yet have any operating system. The node provider needs to install the IC-OS operating system on the node. The detailed procedure can be found in the IC-OS installation runbook articles ([https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Supermicro Installation for SuperMicro], [https://wiki.internetcomputer.org/wiki/IC_OS_Installation_Runbook_-_Dell_Poweredge Installation for Dell Poweredge]).
* The node provider inserts the NitroKey usb stick into the node machine. The NitroKey contains the secret key of corresponding to the node provider's registered public key.
* The node provider then switches on the node to boot the IC-OS operation system, which starts a few processes including orchestrator, crypto and http adapter processes.
* The crypto process finds that it never generated any cryptographic key material before. The crypto process then generates new cryptographic keys. This includes node signing key, NIDKG key, ECDSA key, TLS key, etc.
* The cryptographic key material need to be registered with the NNS registry. For this, the crypto process sends the keys to the orchestrator, which then crafts a message containing the key material, signs the message with the node provider's signing key present in the NitroKey, and sends the message to the NNS registry canister.
* The NNS registry canister creates a record for the new node and stores its cryptographic key material.
* The node is now registered in the Internet Computer, but not yet assigned to any subnet.

== Creating a New Subnet ==
We now describe the process of creating a new subnet. To create a new subnet, one just needs to submit an NNS proposal. The proposal specifies a type (create subnet) and a payload ([[#Payload for 'Create_Subnet' Proposal]]). A few sample proposals to create a new subnet can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/57048 Proposal 57048], [https://dashboard.internetcomputer.org/proposal/49018 Proposal 49018], [https://dashboard.internetcomputer.org/proposal/55730 Proposal 55730]). Anyone who staked their ICP can vote on the proposal within a deadline. After the deadline, if the majority of the voters accept the proposal, then the subnet is created as described below.

The NNS has a canister called "registry", which stores the configuration of the Internet Computer including the configuration of each node and subnet. When the 'Create subnet' proposal is accepted, it automatically calls a method of the NNS Registry with input the payload included in the NNS proposal. The registry canister then creates a new subnet configuration based on the information in the payload, and stores the subnet record. The registry method executed when the 'Create Subnet' NNS proposal is accepted can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/do_create_subnet.rs github].

Each node in the Internet Computer maintains a local copy of the NNS registry. Each node runs a "registry replicator" process which is a part of the orchestrator and is responsible for regularly syncing NNS registry and maintaining the local registry copy. Once every few seconds (can be configured by setting <code>poll_delay_duration_ms</code> variable, which has a default value of 5 seconds), the registry replicator queries the NNS registry for the changes that happened after the recent sync, and applies those changes to the local registry copy. The local registry is stored on the disk and any process running on the node can read the contents of the local registry.

The orchestrator monitors the local registry to see what subnet the node has been assigned to. If the unassigned node has now been assigned to a subnet according to the registry, the orchestrator first downloads the

== Payload for 'Create Subnet' Proposal ==
The payload for the 'Create Subnet' NNS proposal contains the list of parameters to be used by the new subnet. Some of the fields in the payload are as follows. The structure of payloads for every type of NNS proposal can be found on [https://github.com/dfinity/ic/blob/master/rs/registry/canister/canister/registry.did github].
* subnet_type - The type of the subnet to be created. Subnets of different type might exhibit different behavior, e.g. being more restrictive in what operations are allowed or privileged compared to other subnet types. There are a few types of subnets.
** Application subnet - A normal subnet where no restrictions are applied.
** System subnet - A more privileged subnet where certain restrictions are applied, like not charging for cycles or restricting who can create and install canisters on it.
** Verified application subnet - A subnet type that is like application subnets but can have some additional features.
* node_ids - List of node ids of unassigned nodes to be included in the subnet.
* replica_version_id - The version id of the replica software to be installed in the nodes of the subnet.
** A replica image has to be first uploaded to a publicly accessible server.
** Then one needs to create an NNS proposal of type "Elect new binary replica revision" and specify the replica image url and version id in the payload. A sample proposal can be found in the dashboard ([https://dashboard.internetcomputer.org/proposal/92338 proposal 92338]).
** When the proposal is accepted by the majority of voters, the replica version is stored in the NNS registry.
** This replica version id can now be specified in any other NNS proposals including "create subnet".
* Parameters for the P2P layer -
**gossip_max_chunk_size - The maximum chunk size supported on this subnet.
**gossip_retransmission_request_ms - Period for sending retransmission request.
**gossip_receive_check_cache_size - History size for receive check.
**gossip_registry_poll_period_ms - Period for polling the registry for updates.
**gossip_max_duplicity - Max duplicate requests in underutilized networks.
**gossip_max_chunk_wait_ms - Timeout for an outstanding request.
**gossip_pfn_evaluation_period_ms - Period for re evaluating the priority function.
**gossip_max_artifact_streams_per_peer - The maximum number of outstanding request per peer MIN/DEFAULT/MAX.
**advert_best_effort_percentage -
* Parameters for the Consensus layer
** max_block_payload_size - The maximum combined size of the ingress and xnet messages that fit into a block.
** max_ingress_bytes_per_message - The maximum amount of bytes per message. This is a hard cap, which means ingress messages greater than the limit will be dropped.
** max_ingress_messages_per_block - The maximum number of ingress messages allowed per block.
** ingress_bytes_per_block_soft_cap
** initial_notary_delay_millis - Initial delay for notary (in milliseconds), to give time to rank-0 block propagation.
** unit_delay_millis - Unit delay for blockmaker (in milliseconds).
* Parameters for the execution layer
** max_instructions_per_message - The maximum number of instructions a message can execute.
** max_instructions_per_round - The maximum number of instructions a round can execute.
** max_instructions_per_install_code - The maximum number of instructions an "install_code" message can execute.
** max_number_of_canisters - The maximum number of canisters that may be present on the subnet at any given time. A value of 0 is equivalent to setting no limit. This also provides an easy way to maintain compatibility of different versions of replica and registry.
* Parameters related to cryptographic keys
** dkg_dealings_per_block : nat64;
** ecdsa_config : opt EcdsaInitialConfig;
** dkg_interval_length : nat64;
* SSH Access to Subnet
** ssh_readonly_access - The list of public keys whose owners have "readonly" SSH access to all replicas on this subnet, in case it is necessary to perform subnet recovery.
** ssh_backup_access - The list of public keys whose owners have "backup" SSH access to nodes on the NNS subnet to make sure the NNS can be backed up.
* Subnet Features - The list of feature flags to be used for the subnet. This indicates the features that are to be enabled/disabled in the subnet. This includes the following features.
** Canister sandboxing - This feature flag controls whether canister execution happens in sandboxed process or not. It is disabled by default.
** Http requests - This feature flag controls whether canisters of this subnet are capable of performing http(s) requests to the web2.
** Bitcoin testnet feature - Whether or not the subnet is capable of serving requests to the bitcoin testnet canister.
** Bitcoin - Controls whether the bitcoin feature is enabled and which bitcoin network is supported.
* is_halted - If "true", the subnet will be halted: it will no longer create or execute blocks.
* start_as_nns - If set to yes, the subnet starts as a (new) NNS.
* subnet_id_override - This field is optional. If a principal id is specified, then that principal id will be used for the new subnet.

== Executing the 'Create Subnet' Proposal ==